2024-03-08T08:16:05.534 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-08T08:16:05.534 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_jcollin@teuthology --description fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} --name jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi --archive /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510 -- /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/orig.config.yaml 2024-03-08T08:16:05.580 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T08:16:05.668 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510 branch: main description: fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} email: jcollin@redhat.com first_in_suite: false job_id: '7585510' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: lockdep: true mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - missing required features sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-jcollin-test_per-client-perf-counters sha1: f6f80007dcf9e039bb253dd50a92ba3c71c6943a owner: scheduled_jcollin@teuthology priority: 10 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7204 sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 sleep_before_teardown: 0 suite: fs:functional suite_branch: wip-jcollin-test_per-client-perf-counters suite_path: /home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa suite_relpath: qa suite_repo: https://github.com/joscollin/ceph.git suite_sha1: f6f80007dcf9e039bb253dd50a92ba3c71c6943a syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tSPXjRcxJrQoUq/rh0IHWWR+SpHIJgvsXRqThVIwRMImZoQVE9S+8dsvivc5X9Jd/QydIsWWroMlepiN1lIw= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/z/uV83Z6d9tF1DAevK6aqUuf/yu+zfZfYdqeRUiZMOeCd3jl6UOBBo2DMw8+NT1UTHnp3ND54on+Yljk6URQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_admin.TestLabeledPerfCounters teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_08:09:40 tube: smithi user: jcollin verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-08T08:16:05.668 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa; will attempt to use it 2024-03-08T08:16:05.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa/tasks 2024-03-08T08:16:05.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-08T08:16:05.671 INFO:teuthology.task.internal:Checking packages... 2024-03-08T08:16:05.698 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1fb5d10608dccee73edd6ef8a2c7b804a097c324' 2024-03-08T08:16:05.698 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-08T08:16:05.698 INFO:teuthology.packaging:ref: None 2024-03-08T08:16:05.698 INFO:teuthology.packaging:tag: None 2024-03-08T08:16:05.698 INFO:teuthology.packaging:branch: main 2024-03-08T08:16:05.698 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-08T08:16:05.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-08T08:16:06.113 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1940.g639d1827 2024-03-08T08:16:06.114 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-08T08:16:06.121 INFO:teuthology.task.internal:no buildpackages task found 2024-03-08T08:16:06.121 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-08T08:16:06.132 INFO:teuthology.task.internal:Saving configuration 2024-03-08T08:16:06.149 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-08T08:16:06.157 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T08:16:06.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 08:10:46.688364', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tSPXjRcxJrQoUq/rh0IHWWR+SpHIJgvsXRqThVIwRMImZoQVE9S+8dsvivc5X9Jd/QydIsWWroMlepiN1lIw='} 2024-03-08T08:16:06.206 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 08:10:46.689304', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/z/uV83Z6d9tF1DAevK6aqUuf/yu+zfZfYdqeRUiZMOeCd3jl6UOBBo2DMw8+NT1UTHnp3ND54on+Yljk6URQ='} 2024-03-08T08:16:06.206 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-08T08:16:06.216 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-08T08:16:06.216 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-08T08:16:06.216 INFO:teuthology.run_tasks:Running task console_log... 2024-03-08T08:16:06.308 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29ad26a0d0>, signals=[15]) 2024-03-08T08:16:06.308 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-08T08:16:06.317 INFO:teuthology.task.internal:Opening connections... 2024-03-08T08:16:06.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:16:06.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:16:06.433 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:16:06.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:16:06.522 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-08T08:16:06.530 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-03-08T08:16:06.548 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-03-08T08:16:06.548 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-03-08T08:16:06.606 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-03-08T08:16:06.606 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-03-08T08:16:06.606 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-03-08T08:16:06.606 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-03-08T08:16:06.606 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T08:16:06.607 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T08:16:06.608 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-03-08T08:16:06.634 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-03-08T08:16:06.651 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-03-08T08:16:06.652 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-03-08T08:16:06.709 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-03-08T08:16:06.709 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-03-08T08:16:06.709 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T08:16:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T08:16:06.711 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-03-08T08:16:06.732 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-08T08:16:06.742 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-08T08:16:06.750 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-08T08:16:06.750 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-03-08T08:16:06.752 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-03-08T08:16:06.769 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-08T08:16:06.775 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-08T08:16:06.775 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-03-08T08:16:06.809 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-03-08T08:16:06.841 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-08T08:16:06.898 INFO:teuthology.run_tasks:Running task kernel... 2024-03-08T08:16:06.916 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-08T08:16:06.917 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-08T08:16:06.917 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T08:16:06.918 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T08:16:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:16:06.935 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-08T08:16:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:16:06.936 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-08T08:16:06.994 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-361.el9.x86_64 2024-03-08T08:16:06.994 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-361.el9.x86_64 2024-03-08T08:16:06.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-03-08T08:16:06.996 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-03-08T08:16:06.996 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2024-03-08T08:16:06.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-03-08T08:16:09.120 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-08T08:16:09.227 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-08T08:16:10.570 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-08T08:16:10.861 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-08T08:16:12.077 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-08T08:16:22.033 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 725 kB/s | 8.0 MB 00:11 2024-03-08T08:16:26.759 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-03-08T08:16:31.126 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-08T08:16:32.493 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 15 kB 00:00 2024-03-08T08:16:33.812 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-08T08:16:37.998 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 776 kB/s | 19 MB 00:24 2024-03-08T08:16:38.703 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-08T08:16:40.843 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-08T08:16:40.843 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-08T08:16:40.944 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-08T08:16:40.951 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:Install 4 Packages 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 100 M 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 123 M 2024-03-08T08:16:40.952 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-08T08:16:42.413 INFO:teuthology.orchestra.run.smithi156.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.4 MB/s | 6.0 MB 00:01 2024-03-08T08:16:43.105 INFO:teuthology.orchestra.run.smithi156.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 10 MB/s | 21 MB 00:02 2024-03-08T08:16:43.821 INFO:teuthology.orchestra.run.smithi156.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-03-08T08:16:47.696 INFO:teuthology.orchestra.run.smithi156.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.3 MB/s | 33 MB 00:05 2024-03-08T08:16:47.699 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-08T08:16:47.700 INFO:teuthology.orchestra.run.smithi156.stdout:Total 15 MB/s | 100 MB 00:06 2024-03-08T08:16:48.588 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-08T08:16:48.959 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-08T08:16:48.959 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-08T08:16:50.600 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-08T08:16:50.601 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-08T08:16:52.055 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 604 kB/s | 6.1 MB 00:10 2024-03-08T08:16:53.309 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-08T08:16:53.572 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 15 kB 00:00 2024-03-08T08:16:53.969 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-08T08:16:54.164 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:16:56.954 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:16:57.143 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:17:00.250 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 3.1 MB/s | 21 MB 00:06 2024-03-08T08:17:01.883 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:17:02.011 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:05.314 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-08T08:17:06.758 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:07.587 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-08T08:17:07.587 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-08T08:17:07.676 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:17:07.686 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-08T08:17:07.687 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-08T08:17:07.687 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-03-08T08:17:07.687 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:17:07.687 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 100 M 2024-03-08T08:17:07.687 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 123 M 2024-03-08T08:17:07.688 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-08T08:17:08.990 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-core-5.14.0-427.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-03-08T08:17:09.218 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-08T08:17:10.098 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-03-08T08:17:10.690 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 20 MB/s | 33 MB 00:01 2024-03-08T08:17:10.693 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-08T08:17:10.693 INFO:teuthology.orchestra.run.smithi012.stdout:Total 33 MB/s | 100 MB 00:03 2024-03-08T08:17:11.618 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-08T08:17:12.041 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-08T08:17:12.042 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-08T08:17:13.439 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-08T08:17:13.439 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-08T08:17:16.311 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-08T08:17:16.990 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-08T08:17:17.124 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:17:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:17:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:17:25.360 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:17:25.490 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:35.878 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:35.878 INFO:teuthology.orchestra.run.smithi156.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-08T08:17:35.878 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:17:35.962 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:36.401 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:36.401 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-08T08:17:36.402 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:17:36.402 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:17:36.794 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-08T08:17:37.313 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T08:17:37.462 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T08:17:37.463 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T08:17:37.463 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-08T08:17:37.463 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-08T08:17:37.463 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-08T08:17:37.463 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-08T08:17:37.463 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2024-03-08T08:17:37.463 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T08:17:37.584 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T08:17:37.584 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T08:17:37.584 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2024-03-08T08:17:37.628 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2024-03-08T08:17:37.628 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2024-03-08T08:17:37.628 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.06 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 68.el9 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Group : Unspecified 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 8264254 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : builder@centos.org 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2024-03-08T08:17:37.629 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:17:37.630 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2024-03-08T08:17:37.631 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-08T08:17:37.631 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 5.14.0-427.el9.x86_64 2024-03-08T08:17:37.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-08T08:17:38.275 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2024-03-08T08:17:39.634 INFO:teuthology.orchestra.run.smithi156.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-08T08:17:39.659 INFO:teuthology.orchestra.run.smithi156.stderr:done 2024-03-08T08:17:39.662 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-08T08:17:39.677 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.n6Uek39zVT 2024-03-08T08:17:39.678 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n6Uek39zVT 2024-03-08T08:17:39.753 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.n6Uek39zVT 2024-03-08T08:17:39.891 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.n6Uek39zVT is 6KB 2024-03-08T08:17:39.940 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.n6Uek39zVT 2024-03-08T08:17:39.954 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2024-03-08T08:17:40.027 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-08T08:17:40.027 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-08T08:17:40.027 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-08T08:17:40.027 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-08T08:17:40.028 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-08T08:17:40.183 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2024-03-08T08:18:01.116 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:18:01.117 INFO:teuthology.orchestra.run.smithi012.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-08T08:18:01.117 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:18:01.205 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:18:01.668 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:18:01.669 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-08T08:18:01.669 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T08:18:01.669 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:18:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-08T08:18:02.651 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T08:18:02.846 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.846 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T08:18:02.846 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-08T08:18:02.846 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-08T08:18:02.846 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-08T08:18:02.846 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-08T08:18:02.846 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-03-08T08:18:02.847 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T08:18:02.972 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T08:18:02.973 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T08:18:02.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-03-08T08:18:03.013 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-03-08T08:18:03.014 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:18:03.015 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-03-08T08:18:03.016 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-08T08:18:03.016 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-427.el9.x86_64 2024-03-08T08:18:03.016 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-08T08:18:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-03-08T08:18:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-08T08:18:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-03-08T08:18:04.976 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-08T08:18:04.991 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.2CWFXL7cLm 2024-03-08T08:18:04.991 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2CWFXL7cLm 2024-03-08T08:18:05.062 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.2CWFXL7cLm 2024-03-08T08:18:05.204 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.2CWFXL7cLm is 6KB 2024-03-08T08:18:05.254 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.2CWFXL7cLm 2024-03-08T08:18:05.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-03-08T08:18:05.338 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-08T08:18:05.338 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-08T08:18:05.338 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-08T08:18:05.338 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-08T08:18:05.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-08T08:18:05.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-03-08T08:18:10.190 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-08T08:18:10.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:18:10.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:18:28.515 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-08T08:18:35.506 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-08T08:18:35.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:18:35.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:18:37.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:18:37.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:18:40.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-08T08:18:52.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:18:52.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:18:53.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-08T08:19:02.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:19:02.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:19:11.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-08T08:19:21.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-08T08:19:26.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:19:26.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:19:29.742 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-08T08:19:30.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:19:30.206 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-08T08:19:30.207 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-08T08:19:30.220 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-427.el9.x86_64 2024-03-08T08:19:30.220 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-08T08:19:30.221 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-08T08:19:30.221 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-08T08:19:31.221 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-08T08:19:31.222 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-08T08:19:31.309 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-03-08T08:19:31.377 DEBUG:teuthology.parallel:result is None 2024-03-08T08:19:33.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:19:33.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:19:33.683 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-08T08:19:34.271 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:19:34.271 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-08T08:19:34.271 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-08T08:19:34.285 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-427.el9.x86_64 2024-03-08T08:19:34.285 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-08T08:19:34.285 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-08T08:19:34.285 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-08T08:19:35.286 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-08T08:19:35.286 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-08T08:19:35.370 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-03-08T08:19:35.414 DEBUG:teuthology.parallel:result is None 2024-03-08T08:19:35.414 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-08T08:19:35.424 INFO:teuthology.task.internal:Creating test directory... 2024-03-08T08:19:35.425 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T08:19:35.427 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T08:19:35.441 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-08T08:19:35.448 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-08T08:19:35.454 INFO:teuthology.task.internal:Creating archive directory... 2024-03-08T08:19:35.454 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T08:19:35.483 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T08:19:35.510 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-08T08:19:35.517 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-08T08:19:35.517 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T08:19:35.545 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T08:19:35.573 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T08:19:35.583 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T08:19:35.585 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T08:19:35.596 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T08:19:35.597 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-08T08:19:35.604 INFO:teuthology.task.internal:Configuring sudo... 2024-03-08T08:19:35.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T08:19:35.630 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T08:19:35.661 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-08T08:19:35.670 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-08T08:19:35.670 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T08:19:35.699 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T08:19:35.713 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T08:19:35.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T08:19:35.841 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T08:19:35.923 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T08:19:35.965 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:19:35.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T08:19:36.036 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T08:19:36.080 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T08:19:36.154 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T08:19:36.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T08:19:36.247 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:19:36.247 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T08:19:36.330 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-03-08T08:19:36.332 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-03-08T08:19:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T08:19:36.600 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T08:19:36.914 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-08T08:19:36.922 INFO:teuthology.task.internal:Starting timer... 2024-03-08T08:19:36.923 INFO:teuthology.run_tasks:Running task pcp... 2024-03-08T08:19:36.934 INFO:teuthology.run_tasks:Running task selinux... 2024-03-08T08:19:36.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-03-08T08:19:37.002 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-03-08T08:19:37.004 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-03-08T08:19:37.098 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-03-08T08:19:37.104 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-08T08:19:37.104 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-03-08T08:19:37.120 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-03-08T08:19:37.121 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-03-08T08:19:37.154 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-03-08T08:19:37.154 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-03-08T08:19:37.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-08T08:19:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:19:37.193 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-08T08:19:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:19:37.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-08T08:19:37.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-03-08T08:19:37.278 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-03-08T08:19:37.313 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-08T08:19:37.322 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-08T08:19:37.340 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-08T08:19:37.342 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-08T08:23:51.519 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-03-08T08:23:51.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:23:51.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:23:51.622 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-08T08:23:51.701 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-08T08:23:51.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:23:51.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T08:23:51.789 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-08T08:23:51.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-08T08:23:51.867 INFO:teuthology.run_tasks:Running task clock... 2024-03-08T08:23:51.877 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-08T08:23:51.878 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T08:23:51.878 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T08:23:51.880 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T08:23:51.881 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T08:23:51.911 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T08:23:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T08:23:51.938 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T08:23:51.956 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T08:23:51.960 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-03-08T08:23:51.974 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-03-08T08:23:51.986 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-03-08T08:23:51.991 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T08:23:52.000 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-03-08T08:23:52.008 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T08:23:52.017 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T08:23:52.033 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T08:23:52.069 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-08T08:23:52.072 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T08:23:52.072 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-08T08:23:52.072 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.073 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.073 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.073 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.085 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.089 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:23:52.090 INFO:teuthology.run_tasks:Running task install... 2024-03-08T08:23:52.098 DEBUG:teuthology.task.install:project ceph 2024-03-08T08:23:52.099 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}} 2024-03-08T08:23:52.099 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'} 2024-03-08T08:23:52.099 INFO:teuthology.task.install:Using flavor: default 2024-03-08T08:23:52.105 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-08T08:23:52.105 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-08T08:23:52.105 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-03-08T08:23:52.106 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-08T08:23:52.107 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-03-08T08:23:52.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-08T08:23:52.464 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/ 2024-03-08T08:23:52.464 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1670.g1fb5d106 2024-03-08T08:23:52.467 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/ 2024-03-08T08:23:52.468 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1670.g1fb5d106 2024-03-08T08:23:52.657 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T08:23:52.657 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:23:52.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T08:23:52.687 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T08:23:52.687 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1fb5d10608dccee73edd6ef8a2c7b804a097c324/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T08:23:52.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T08:23:52.780 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T08:23:52.780 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:23:52.780 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T08:23:52.809 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T08:23:52.810 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1fb5d10608dccee73edd6ef8a2c7b804a097c324/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T08:23:52.838 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T08:23:52.878 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T08:23:52.906 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-03-08T08:23:52.907 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-03-08T08:23:52.959 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T08:23:53.026 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-03-08T08:23:53.028 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-03-08T08:23:53.152 INFO:teuthology.orchestra.run.smithi012.stdout:74 files removed 2024-03-08T08:23:53.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T08:23:53.269 INFO:teuthology.orchestra.run.smithi156.stdout:74 files removed 2024-03-08T08:23:53.296 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T08:23:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 102 kB/s | 82 kB 00:00 2024-03-08T08:23:54.612 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 82 kB/s | 82 kB 00:00 2024-03-08T08:23:54.962 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-08T08:23:55.137 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-08T08:23:55.705 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-08T08:23:55.763 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-08T08:23:56.632 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-08T08:23:56.690 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-08T08:23:58.290 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-08T08:23:58.565 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-08T08:24:02.825 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-08T08:24:03.821 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:04 2024-03-08T08:24:05.015 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-08T08:24:05.814 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-08T08:24:12.056 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-03-08T08:24:13.473 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 106 kB/s | 15 kB 00:00 2024-03-08T08:24:19.306 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 582 kB/s | 6.1 MB 00:10 2024-03-08T08:24:22.373 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 8.8 kB/s | 15 kB 00:01 2024-03-08T08:24:25.743 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-03-08T08:24:30.558 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-08T08:24:33.071 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T08:24:33.072 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T08:24:33.086 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T08:24:33.087 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T08:24:33.088 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T08:24:33.089 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T08:24:33.091 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T08:24:33.093 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T08:24:33.094 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T08:24:33.175 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 6.6 k 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.3 M 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 887 k 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 145 k 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 1.5 M 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 138 k 2024-03-08T08:24:33.196 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 1.8 M 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 7.4 M 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 49 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 9.8 M 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 47 M 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 264 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 748 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 220 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 27 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 18 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 31 k 2024-03-08T08:24:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 745 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 127 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 162 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 320 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 298 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 99 k 2024-03-08T08:24:33.198 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 85 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 170 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T08:24:33.199 INFO:teuthology.orchestra.run.smithi156.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.4 M 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T08:24:33.200 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 20 M 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 31 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 2.1 M 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 248 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 4.7 M 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 17 M 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 15 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 25 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T08:24:33.201 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 163 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T08:24:33.202 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 501 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.1 M 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T08:24:33.203 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 45 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 131 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T08:24:33.204 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T08:24:33.205 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T08:24:33.206 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T08:24:33.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T08:24:33.208 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout:Install 150 Packages 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 24 Packages 2024-03-08T08:24:33.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:24:33.210 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 223 M 2024-03-08T08:24:33.210 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-08T08:24:33.951 INFO:teuthology.orchestra.run.smithi156.stdout:(1/174): ceph-19.0.0-1670.g1fb5d106.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-08T08:24:34.159 INFO:teuthology.orchestra.run.smithi156.stdout:(2/174): ceph-fuse-19.0.0-1670.g1fb5d106.el9.x8 4.2 MB/s | 887 kB 00:00 2024-03-08T08:24:34.201 INFO:teuthology.orchestra.run.smithi156.stdout:(3/174): ceph-immutable-object-cache-19.0.0-167 3.4 MB/s | 145 kB 00:00 2024-03-08T08:24:34.385 INFO:teuthology.orchestra.run.smithi156.stdout:(4/174): ceph-mds-19.0.0-1670.g1fb5d106.el9.x86 12 MB/s | 2.1 MB 00:00 2024-03-08T08:24:34.577 INFO:teuthology.orchestra.run.smithi156.stdout:(5/174): ceph-mgr-19.0.0-1670.g1fb5d106.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-03-08T08:24:35.244 INFO:teuthology.orchestra.run.smithi156.stdout:(6/174): ceph-mon-19.0.0-1670.g1fb5d106.el9.x86 7.0 MB/s | 4.7 MB 00:00 2024-03-08T08:24:36.144 INFO:teuthology.orchestra.run.smithi156.stdout:(7/174): ceph-base-19.0.0-1670.g1fb5d106.el9.x8 2.3 MB/s | 5.3 MB 00:02 2024-03-08T08:24:37.852 INFO:teuthology.orchestra.run.smithi156.stdout:(8/174): ceph-radosgw-19.0.0-1670.g1fb5d106.el9 5.7 MB/s | 9.8 MB 00:01 2024-03-08T08:24:37.952 INFO:teuthology.orchestra.run.smithi156.stdout:(9/174): ceph-selinux-19.0.0-1670.g1fb5d106.el9 252 kB/s | 25 kB 00:00 2024-03-08T08:24:40.044 INFO:teuthology.orchestra.run.smithi156.stdout:(10/174): ceph-osd-19.0.0-1670.g1fb5d106.el9.x8 3.5 MB/s | 17 MB 00:04 2024-03-08T08:24:40.111 INFO:teuthology.orchestra.run.smithi156.stdout:(11/174): cephfs-mirror-19.0.0-1670.g1fb5d106.e 3.2 MB/s | 220 kB 00:00 2024-03-08T08:24:40.153 INFO:teuthology.orchestra.run.smithi156.stdout:(12/174): cephfs-shell-19.0.0-1670.g1fb5d106.el 649 kB/s | 27 kB 00:00 2024-03-08T08:24:40.195 INFO:teuthology.orchestra.run.smithi156.stdout:(13/174): libcephfs-devel-19.0.0-1670.g1fb5d106 751 kB/s | 31 kB 00:00 2024-03-08T08:24:40.312 INFO:teuthology.orchestra.run.smithi156.stdout:(14/174): libcephfs2-19.0.0-1670.g1fb5d106.el9. 6.3 MB/s | 745 kB 00:00 2024-03-08T08:24:40.362 INFO:teuthology.orchestra.run.smithi156.stdout:(15/174): libcephsqlite-19.0.0-1670.g1fb5d106.e 3.2 MB/s | 163 kB 00:00 2024-03-08T08:24:40.404 INFO:teuthology.orchestra.run.smithi156.stdout:(16/174): librados-devel-19.0.0-1670.g1fb5d106. 3.0 MB/s | 127 kB 00:00 2024-03-08T08:24:40.513 INFO:teuthology.orchestra.run.smithi156.stdout:(17/174): libradosstriper1-19.0.0-1670.g1fb5d10 4.5 MB/s | 501 kB 00:00 2024-03-08T08:24:41.063 INFO:teuthology.orchestra.run.smithi156.stdout:(18/174): librgw2-19.0.0-1670.g1fb5d106.el9.x86 9.3 MB/s | 5.1 MB 00:00 2024-03-08T08:24:41.105 INFO:teuthology.orchestra.run.smithi156.stdout:(19/174): python3-ceph-argparse-19.0.0-1670.g1f 1.1 MB/s | 45 kB 00:00 2024-03-08T08:24:41.155 INFO:teuthology.orchestra.run.smithi156.stdout:(20/174): python3-ceph-common-19.0.0-1670.g1fb5 2.6 MB/s | 131 kB 00:00 2024-03-08T08:24:41.205 INFO:teuthology.orchestra.run.smithi156.stdout:(21/174): python3-cephfs-19.0.0-1670.g1fb5d106. 3.2 MB/s | 162 kB 00:00 2024-03-08T08:24:41.297 INFO:teuthology.orchestra.run.smithi156.stdout:(22/174): python3-rados-19.0.0-1670.g1fb5d106.e 3.4 MB/s | 320 kB 00:00 2024-03-08T08:24:41.381 INFO:teuthology.orchestra.run.smithi156.stdout:(23/174): python3-rbd-19.0.0-1670.g1fb5d106.el9 3.5 MB/s | 298 kB 00:00 2024-03-08T08:24:41.440 INFO:teuthology.orchestra.run.smithi156.stdout:(24/174): python3-rgw-19.0.0-1670.g1fb5d106.el9 1.7 MB/s | 99 kB 00:00 2024-03-08T08:24:41.498 INFO:teuthology.orchestra.run.smithi156.stdout:(25/174): rbd-fuse-19.0.0-1670.g1fb5d106.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-08T08:24:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:(26/174): rbd-mirror-19.0.0-1670.g1fb5d106.el9. 4.8 MB/s | 3.1 MB 00:00 2024-03-08T08:24:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:(27/174): rbd-nbd-19.0.0-1670.g1fb5d106.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-08T08:24:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1670.g 469 kB/s | 31 kB 00:00 2024-03-08T08:24:42.366 INFO:teuthology.orchestra.run.smithi156.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1670.g1fb5d10 1.6 MB/s | 138 kB 00:00 2024-03-08T08:24:42.917 INFO:teuthology.orchestra.run.smithi156.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1670.g1fb5d 3.2 MB/s | 1.8 MB 00:00 2024-03-08T08:24:44.792 INFO:teuthology.orchestra.run.smithi156.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-03-08T08:24:44.950 INFO:teuthology.orchestra.run.smithi156.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1670.g1f 1.5 MB/s | 248 kB 00:00 2024-03-08T08:24:45.042 INFO:teuthology.orchestra.run.smithi156.stdout:(33/174): ceph-mgr-rook-19.0.0-1670.g1fb5d106.e 538 kB/s | 49 kB 00:00 2024-03-08T08:24:45.126 INFO:teuthology.orchestra.run.smithi156.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1670.g1 184 kB/s | 15 kB 00:00 2024-03-08T08:24:45.284 INFO:teuthology.orchestra.run.smithi156.stdout:(35/174): ceph-volume-19.0.0-1670.g1fb5d106.el9 1.6 MB/s | 264 kB 00:00 2024-03-08T08:24:45.585 INFO:teuthology.orchestra.run.smithi156.stdout:(36/174): cephadm-19.0.0-1670.g1fb5d106.el9.noa 2.4 MB/s | 748 kB 00:00 2024-03-08T08:24:45.685 INFO:teuthology.orchestra.run.smithi156.stdout:(37/174): cephfs-top-19.0.0-1670.g1fb5d106.el9. 183 kB/s | 18 kB 00:00 2024-03-08T08:24:46.110 INFO:teuthology.orchestra.run.smithi156.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-08T08:24:46.285 INFO:teuthology.orchestra.run.smithi156.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 5.1 MB/s | 920 kB 00:00 2024-03-08T08:24:46.336 INFO:teuthology.orchestra.run.smithi156.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-08T08:24:46.394 INFO:teuthology.orchestra.run.smithi156.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-08T08:24:46.436 INFO:teuthology.orchestra.run.smithi156.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 762 kB/s | 32 kB 00:00 2024-03-08T08:24:46.611 INFO:teuthology.orchestra.run.smithi156.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-08T08:24:46.662 INFO:teuthology.orchestra.run.smithi156.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 342 kB/s | 17 kB 00:00 2024-03-08T08:24:46.712 INFO:teuthology.orchestra.run.smithi156.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-08T08:24:46.771 INFO:teuthology.orchestra.run.smithi156.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-08T08:24:46.821 INFO:teuthology.orchestra.run.smithi156.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-08T08:24:46.888 INFO:teuthology.orchestra.run.smithi156.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-08T08:24:46.946 INFO:teuthology.orchestra.run.smithi156.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-08T08:24:46.997 INFO:teuthology.orchestra.run.smithi156.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 927 kB/s | 46 kB 00:00 2024-03-08T08:24:47.055 INFO:teuthology.orchestra.run.smithi156.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-08T08:24:47.106 INFO:teuthology.orchestra.run.smithi156.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-08T08:24:47.164 INFO:teuthology.orchestra.run.smithi156.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-08T08:24:47.215 INFO:teuthology.orchestra.run.smithi156.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-08T08:24:47.457 INFO:teuthology.orchestra.run.smithi156.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-08T08:24:47.540 INFO:teuthology.orchestra.run.smithi156.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 18 MB/s | 1.5 MB 00:00 2024-03-08T08:24:47.607 INFO:teuthology.orchestra.run.smithi156.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-08T08:24:47.658 INFO:teuthology.orchestra.run.smithi156.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 2.6 MB/s | 134 kB 00:00 2024-03-08T08:24:47.883 INFO:teuthology.orchestra.run.smithi156.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-08T08:24:47.950 INFO:teuthology.orchestra.run.smithi156.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-08T08:24:48.000 INFO:teuthology.orchestra.run.smithi156.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-08T08:24:48.234 INFO:teuthology.orchestra.run.smithi156.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-08T08:24:48.284 INFO:teuthology.orchestra.run.smithi156.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 24 kB 00:00 2024-03-08T08:24:48.335 INFO:teuthology.orchestra.run.smithi156.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-08T08:24:48.385 INFO:teuthology.orchestra.run.smithi156.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-08T08:24:48.427 INFO:teuthology.orchestra.run.smithi156.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 759 kB/s | 32 kB 00:00 2024-03-08T08:24:48.628 INFO:teuthology.orchestra.run.smithi156.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-08T08:24:48.677 INFO:teuthology.orchestra.run.smithi156.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-08T08:24:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-08T08:24:48.778 INFO:teuthology.orchestra.run.smithi156.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-08T08:24:48.828 INFO:teuthology.orchestra.run.smithi156.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-08T08:24:48.878 INFO:teuthology.orchestra.run.smithi156.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-08T08:24:48.929 INFO:teuthology.orchestra.run.smithi156.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-08T08:24:49.004 INFO:teuthology.orchestra.run.smithi156.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-08T08:24:49.062 INFO:teuthology.orchestra.run.smithi156.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 809 kB/s | 47 kB 00:00 2024-03-08T08:24:49.121 INFO:teuthology.orchestra.run.smithi156.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-03-08T08:24:49.655 INFO:teuthology.orchestra.run.smithi156.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-08T08:24:49.705 INFO:teuthology.orchestra.run.smithi156.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-08T08:24:49.747 INFO:teuthology.orchestra.run.smithi156.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 984 kB/s | 41 kB 00:00 2024-03-08T08:24:49.814 INFO:teuthology.orchestra.run.smithi156.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-08T08:24:49.864 INFO:teuthology.orchestra.run.smithi156.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-08T08:24:49.914 INFO:teuthology.orchestra.run.smithi156.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-08T08:24:49.965 INFO:teuthology.orchestra.run.smithi156.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-08T08:24:50.024 INFO:teuthology.orchestra.run.smithi156.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-08T08:24:50.082 INFO:teuthology.orchestra.run.smithi156.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-08T08:24:50.174 INFO:teuthology.orchestra.run.smithi156.stdout:(86/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-08T08:24:50.233 INFO:teuthology.orchestra.run.smithi156.stdout:(87/174): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-08T08:24:50.283 INFO:teuthology.orchestra.run.smithi156.stdout:(88/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-08T08:24:50.359 INFO:teuthology.orchestra.run.smithi156.stdout:(89/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-08T08:24:51.084 INFO:teuthology.orchestra.run.smithi156.stdout:(90/174): jq-1.6-16.el9.x86_64.rpm 258 kB/s | 187 kB 00:00 2024-03-08T08:24:51.434 INFO:teuthology.orchestra.run.smithi156.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-08T08:24:51.576 INFO:teuthology.orchestra.run.smithi156.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-08T08:24:51.701 INFO:teuthology.orchestra.run.smithi156.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-08T08:24:51.785 INFO:teuthology.orchestra.run.smithi156.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 259 kB/s | 21 kB 00:00 2024-03-08T08:24:51.836 INFO:teuthology.orchestra.run.smithi156.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-03-08T08:24:52.036 INFO:teuthology.orchestra.run.smithi156.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 21 MB/s | 4.1 MB 00:00 2024-03-08T08:24:52.086 INFO:teuthology.orchestra.run.smithi156.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-08T08:24:52.128 INFO:teuthology.orchestra.run.smithi156.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-03-08T08:24:52.170 INFO:teuthology.orchestra.run.smithi156.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-08T08:24:52.212 INFO:teuthology.orchestra.run.smithi156.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 834 kB/s | 35 kB 00:00 2024-03-08T08:24:52.262 INFO:teuthology.orchestra.run.smithi156.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-08T08:24:52.304 INFO:teuthology.orchestra.run.smithi156.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-08T08:24:52.479 INFO:teuthology.orchestra.run.smithi156.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-08T08:24:52.663 INFO:teuthology.orchestra.run.smithi156.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-03-08T08:24:52.888 INFO:teuthology.orchestra.run.smithi156.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-03-08T08:24:52.938 INFO:teuthology.orchestra.run.smithi156.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 6.0 MB/s | 309 kB 00:00 2024-03-08T08:24:53.005 INFO:teuthology.orchestra.run.smithi156.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-08T08:24:53.155 INFO:teuthology.orchestra.run.smithi156.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-03-08T08:24:53.214 INFO:teuthology.orchestra.run.smithi156.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-08T08:24:53.265 INFO:teuthology.orchestra.run.smithi156.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-03-08T08:24:53.307 INFO:teuthology.orchestra.run.smithi156.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-08T08:24:53.365 INFO:teuthology.orchestra.run.smithi156.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-08T08:24:53.425 INFO:teuthology.orchestra.run.smithi156.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-08T08:24:53.541 INFO:teuthology.orchestra.run.smithi156.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-08T08:24:53.600 INFO:teuthology.orchestra.run.smithi156.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 749 kB/s | 43 kB 00:00 2024-03-08T08:24:53.675 INFO:teuthology.orchestra.run.smithi156.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 429 kB/s | 32 kB 00:00 2024-03-08T08:24:53.767 INFO:teuthology.orchestra.run.smithi156.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-08T08:24:53.851 INFO:teuthology.orchestra.run.smithi156.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-08T08:24:53.943 INFO:teuthology.orchestra.run.smithi156.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-08T08:24:54.051 INFO:teuthology.orchestra.run.smithi156.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-08T08:24:54.135 INFO:teuthology.orchestra.run.smithi156.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 424 kB/s | 35 kB 00:00 2024-03-08T08:24:54.194 INFO:teuthology.orchestra.run.smithi156.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-08T08:24:54.369 INFO:teuthology.orchestra.run.smithi156.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-08T08:24:54.436 INFO:teuthology.orchestra.run.smithi156.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-08T08:24:54.486 INFO:teuthology.orchestra.run.smithi156.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-08T08:24:54.570 INFO:teuthology.orchestra.run.smithi156.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-03-08T08:24:54.612 INFO:teuthology.orchestra.run.smithi156.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 559 kB/s | 23 kB 00:00 2024-03-08T08:24:54.654 INFO:teuthology.orchestra.run.smithi156.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 466 kB/s | 19 kB 00:00 2024-03-08T08:24:54.696 INFO:teuthology.orchestra.run.smithi156.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 471 kB/s | 20 kB 00:00 2024-03-08T08:24:54.829 INFO:teuthology.orchestra.run.smithi156.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 7.7 MB/s | 1.0 MB 00:00 2024-03-08T08:24:54.888 INFO:teuthology.orchestra.run.smithi156.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-03-08T08:24:54.947 INFO:teuthology.orchestra.run.smithi156.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-08T08:24:55.022 INFO:teuthology.orchestra.run.smithi156.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 771 kB/s | 58 kB 00:00 2024-03-08T08:24:55.122 INFO:teuthology.orchestra.run.smithi156.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-08T08:24:55.264 INFO:teuthology.orchestra.run.smithi156.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-03-08T08:24:55.331 INFO:teuthology.orchestra.run.smithi156.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-08T08:24:55.381 INFO:teuthology.orchestra.run.smithi156.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-03-08T08:24:55.440 INFO:teuthology.orchestra.run.smithi156.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-03-08T08:24:55.499 INFO:teuthology.orchestra.run.smithi156.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-08T08:24:55.566 INFO:teuthology.orchestra.run.smithi156.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 886 kB/s | 59 kB 00:00 2024-03-08T08:24:55.641 INFO:teuthology.orchestra.run.smithi156.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 478 kB/s | 36 kB 00:00 2024-03-08T08:24:55.700 INFO:teuthology.orchestra.run.smithi156.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 877 kB/s | 51 kB 00:00 2024-03-08T08:24:55.775 INFO:teuthology.orchestra.run.smithi156.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-08T08:24:55.859 INFO:teuthology.orchestra.run.smithi156.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-08T08:24:55.934 INFO:teuthology.orchestra.run.smithi156.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-08T08:24:56.001 INFO:teuthology.orchestra.run.smithi156.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 301 kB/s | 20 kB 00:00 2024-03-08T08:24:56.051 INFO:teuthology.orchestra.run.smithi156.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-08T08:24:56.110 INFO:teuthology.orchestra.run.smithi156.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-08T08:24:56.569 INFO:teuthology.orchestra.run.smithi156.stdout:(149/174): librados2-19.0.0-1670.g1fb5d106.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-08T08:24:56.930 INFO:teuthology.orchestra.run.smithi156.stdout:(150/174): librbd1-19.0.0-1670.g1fb5d106.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-03-08T08:24:57.044 INFO:teuthology.orchestra.run.smithi156.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-08T08:24:58.128 INFO:teuthology.orchestra.run.smithi156.stdout:(152/174): ceph-common-19.0.0-1670.g1fb5d106.el 838 kB/s | 20 MB 00:24 2024-03-08T08:25:01.027 INFO:teuthology.orchestra.run.smithi156.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 588 kB/s | 2.3 MB 00:03 2024-03-08T08:25:01.153 INFO:teuthology.orchestra.run.smithi156.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 339 kB/s | 42 kB 00:00 2024-03-08T08:25:01.770 INFO:teuthology.orchestra.run.smithi156.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 402 kB/s | 248 kB 00:00 2024-03-08T08:25:02.012 INFO:teuthology.orchestra.run.smithi156.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 295 kB/s | 71 kB 00:00 2024-03-08T08:25:02.137 INFO:teuthology.orchestra.run.smithi156.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 292 kB/s | 36 kB 00:00 2024-03-08T08:25:02.521 INFO:teuthology.orchestra.run.smithi156.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 502 kB/s | 192 kB 00:00 2024-03-08T08:25:02.929 INFO:teuthology.orchestra.run.smithi156.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 626 kB/s | 256 kB 00:00 2024-03-08T08:25:03.046 INFO:teuthology.orchestra.run.smithi156.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 198 kB/s | 23 kB 00:00 2024-03-08T08:25:03.188 INFO:teuthology.orchestra.run.smithi156.stdout:(161/174): NetworkManager-libnm-1.46.0-1.el9.x8 368 kB/s | 1.8 MB 00:05 2024-03-08T08:25:03.313 INFO:teuthology.orchestra.run.smithi156.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 406 kB/s | 108 kB 00:00 2024-03-08T08:25:03.497 INFO:teuthology.orchestra.run.smithi156.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 502 kB/s | 155 kB 00:00 2024-03-08T08:25:03.547 INFO:teuthology.orchestra.run.smithi156.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 580 kB/s | 136 kB 00:00 2024-03-08T08:25:03.623 INFO:teuthology.orchestra.run.smithi156.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 508 kB/s | 64 kB 00:00 2024-03-08T08:25:03.656 INFO:teuthology.orchestra.run.smithi156.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 264 kB/s | 29 kB 00:00 2024-03-08T08:25:05.348 INFO:teuthology.orchestra.run.smithi156.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 718 kB/s | 1.2 MB 00:01 2024-03-08T08:25:05.667 INFO:teuthology.orchestra.run.smithi156.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.1 MB/s | 2.2 MB 00:02 2024-03-08T08:25:05.984 INFO:teuthology.orchestra.run.smithi156.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 1.4 MB/s | 465 kB 00:00 2024-03-08T08:25:06.126 INFO:teuthology.orchestra.run.smithi156.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 220 kB/s | 31 kB 00:00 2024-03-08T08:25:07.801 INFO:teuthology.orchestra.run.smithi156.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 656 kB/s | 1.1 MB 00:01 2024-03-08T08:25:07.877 INFO:teuthology.orchestra.run.smithi156.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 992 kB/s | 74 kB 00:00 2024-03-08T08:25:08.360 INFO:teuthology.orchestra.run.smithi156.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 776 kB/s | 2.3 MB 00:03 2024-03-08T08:25:22.759 INFO:teuthology.orchestra.run.smithi156.stdout:(174/174): ceph-test-19.0.0-1670.g1fb5d106.el9. 1.1 MB/s | 47 MB 00:44 2024-03-08T08:25:22.762 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-08T08:25:22.762 INFO:teuthology.orchestra.run.smithi156.stdout:Total 4.5 MB/s | 223 MB 00:49 2024-03-08T08:25:23.209 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-08T08:25:23.520 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-08T08:25:23.520 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-08T08:25:25.545 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-08T08:25:25.546 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-08T08:25:32.487 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-08T08:25:32.612 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T08:25:32.648 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T08:25:32.693 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T08:25:32.703 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T08:25:32.743 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T08:25:32.812 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T08:25:32.847 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T08:25:32.872 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T08:25:32.912 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T08:25:33.242 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T08:25:33.290 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:25:33.422 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:25:33.449 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:25:33.524 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:25:33.556 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/198 2024-03-08T08:25:33.602 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T08:25:33.793 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T08:25:33.843 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T08:25:33.871 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:25:33.957 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:25:33.980 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 17/198 2024-03-08T08:25:34.037 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 17/198 2024-03-08T08:25:34.068 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T08:25:34.090 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T08:25:34.105 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T08:25:34.240 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T08:25:34.272 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T08:25:34.295 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T08:25:34.479 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T08:25:34.605 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T08:25:34.665 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T08:25:34.776 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:25:34.841 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:25:35.053 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:25:35.162 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T08:25:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T08:25:35.304 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T08:25:35.332 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T08:25:35.376 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T08:25:35.450 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T08:25:35.503 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T08:25:35.542 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T08:25:35.616 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T08:25:35.652 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T08:25:35.695 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T08:25:35.728 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T08:25:35.756 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T08:25:35.835 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T08:25:35.869 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9. 39/198 2024-03-08T08:25:35.920 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el 40/198 2024-03-08T08:25:35.956 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_6 41/198 2024-03-08T08:25:36.021 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T08:25:36.073 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T08:25:36.148 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T08:25:36.286 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T08:25:36.343 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T08:25:36.886 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T08:25:36.972 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T08:25:37.050 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T08:25:37.099 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T08:25:37.123 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T08:25:37.157 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T08:25:37.205 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T08:25:37.243 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_6 53/198 2024-03-08T08:25:37.273 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T08:25:37.545 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T08:25:37.716 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T08:25:37.776 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T08:25:37.819 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T08:25:37.950 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T08:25:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T08:25:38.062 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T08:25:38.144 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T08:25:38.174 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T08:25:38.210 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T08:25:38.258 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T08:25:38.293 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T08:25:38.336 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T08:25:38.378 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T08:25:38.515 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T08:25:38.567 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T08:25:38.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T08:25:38.639 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T08:25:38.735 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T08:25:38.773 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T08:25:38.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T08:25:38.839 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T08:25:38.863 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T08:25:38.951 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T08:25:39.082 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T08:25:39.190 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T08:25:39.230 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T08:25:39.279 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T08:25:39.336 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T08:25:39.525 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T08:25:39.585 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T08:25:39.727 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T08:25:39.775 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T08:25:39.807 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T08:25:39.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T08:25:39.866 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T08:25:39.900 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T08:25:39.939 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T08:25:39.988 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T08:25:40.018 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T08:25:40.095 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T08:25:40.128 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T08:25:40.162 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T08:25:40.203 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T08:25:40.238 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T08:25:40.329 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T08:25:40.411 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T08:25:40.466 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T08:25:40.497 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T08:25:40.543 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T08:25:40.643 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T08:25:40.698 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T08:25:41.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T08:25:41.488 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T08:25:41.521 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T08:25:41.570 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T08:25:41.660 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T08:25:41.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T08:25:41.883 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T08:25:41.920 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T08:25:41.998 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T08:25:42.661 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T08:25:42.777 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T08:25:44.024 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T08:25:44.097 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T08:25:44.128 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T08:25:44.159 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T08:25:44.283 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T08:25:44.760 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T08:25:44.792 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 124/198 2024-03-08T08:25:44.875 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 124/198 2024-03-08T08:25:44.905 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 125/198 2024-03-08T08:25:44.942 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T08:25:45.394 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T08:25:45.420 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 128/198 2024-03-08T08:25:45.491 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 128/198 2024-03-08T08:25:45.514 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 129/198 2024-03-08T08:25:47.661 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:25:47.693 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:25:48.249 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:25:48.289 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 131/198 2024-03-08T08:25:48.374 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 131/198 2024-03-08T08:25:48.374 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T08:25:48.374 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T08:25:48.374 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:25:48.407 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 132/198 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 132/198 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-03-08T08:26:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-03-08T08:26:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-03-08T08:26:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:00.279 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 133/198 2024-03-08T08:26:00.294 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 133/198 2024-03-08T08:26:00.294 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:00.295 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T08:26:00.295 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:01.790 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 134/198 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 134/198 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T08:26:01.815 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:02.033 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 135/198 2024-03-08T08:26:02.055 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 135/198 2024-03-08T08:26:02.055 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:02.055 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T08:26:02.056 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T08:26:02.056 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T08:26:02.056 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:02.502 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 136/198 2024-03-08T08:26:02.524 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 136/198 2024-03-08T08:26:02.525 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:02.525 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T08:26:02.525 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T08:26:02.525 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T08:26:02.525 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:02.612 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T08:26:03.056 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 138/198 2024-03-08T08:26:03.085 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 138/198 2024-03-08T08:26:03.117 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.e 139/198 2024-03-08T08:26:03.177 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106. 140/198 2024-03-08T08:26:03.200 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 141/198 2024-03-08T08:26:03.430 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 141/198 2024-03-08T08:26:03.460 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 142/198 2024-03-08T08:26:04.425 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 142/198 2024-03-08T08:26:04.470 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 143/198 2024-03-08T08:26:04.571 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 143/198 2024-03-08T08:26:04.740 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el 144/198 2024-03-08T08:26:04.771 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 145/198 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 145/198 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T08:26:04.793 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:04.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 146/198 2024-03-08T08:26:04.849 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 146/198 2024-03-08T08:26:04.900 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 147/198 2024-03-08T08:26:05.765 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T08:26:05.790 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 149/198 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 149/198 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T08:26:05.813 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:05.850 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 150/198 2024-03-08T08:26:05.871 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 150/198 2024-03-08T08:26:05.871 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:05.871 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T08:26:05.871 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:05.917 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:26:05.939 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:26:05.939 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:05.939 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T08:26:05.939 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T08:26:05.939 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T08:26:05.940 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:06.251 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 152/198 2024-03-08T08:26:06.273 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 152/198 2024-03-08T08:26:06.273 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:26:06.273 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T08:26:06.274 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T08:26:06.274 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T08:26:06.274 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:10.801 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 153/198 2024-03-08T08:26:10.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 154/198 2024-03-08T08:26:10.877 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 155/198 2024-03-08T08:26:10.904 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T08:26:11.091 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 157/198 2024-03-08T08:26:11.203 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T08:26:11.244 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 159/198 2024-03-08T08:26:11.299 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T08:26:11.353 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T08:26:11.396 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T08:26:11.420 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_ 163/198 2024-03-08T08:26:11.461 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T08:26:11.491 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T08:26:11.547 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T08:26:11.605 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T08:26:11.631 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T08:26:12.204 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:26:12.248 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:26:12.268 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:26:12.669 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 177 kB/s | 21 MB 01:59 2024-03-08T08:26:13.120 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 170/198 2024-03-08T08:26:13.194 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T08:26:13.235 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T08:26:13.318 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T08:26:13.348 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T08:26:13.372 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T08:26:13.398 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T08:26:13.424 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T08:26:13.424 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T08:26:13.513 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T08:26:13.542 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T08:26:13.542 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T08:26:13.627 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T08:26:13.658 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T08:26:13.686 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T08:26:13.711 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T08:26:13.737 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T08:26:13.780 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T08:26:13.818 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T08:26:13.866 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T08:26:13.891 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T08:26:13.917 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T08:26:13.942 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T08:26:13.968 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T08:26:13.974 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:26:13.974 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:26:14.154 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:26:14.154 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T08:26:14.237 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T08:26:14.261 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T08:26:14.287 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T08:26:14.320 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T08:26:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T08:26:14.420 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T08:26:17.877 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-08T08:26:19.326 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T08:26:19.326 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/198 2024-03-08T08:26:19.326 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/198 2024-03-08T08:26:19.326 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/198 2024-03-08T08:26:19.326 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 5/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 8/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 13/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 14/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_ 15/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:26:19.327 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 17/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_6 18/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 19/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 20/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el 21/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9. 22/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_6 23/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 24/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 25/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 27/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 28/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/198 2024-03-08T08:26:19.328 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106. 30/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 31/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 32/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 33/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el 34/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 35/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.e 36/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 37/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 38/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 39/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T08:26:19.329 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T08:26:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T08:26:19.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T08:26:19.332 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T08:26:19.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T08:26:19.334 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T08:26:19.335 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T08:26:19.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T08:26:19.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:26:19.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 153/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T08:26:19.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T08:26:19.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T08:26:19.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T08:26:19.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T08:26:20.441 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T08:26:20.442 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T08:26:20.456 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T08:26:20.458 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T08:26:20.459 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T08:26:20.460 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T08:26:20.462 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T08:26:20.465 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T08:26:20.466 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T08:26:20.554 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 6.6 k 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.3 M 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 887 k 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 145 k 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 1.5 M 2024-03-08T08:26:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 138 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 1.8 M 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 7.4 M 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 49 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 9.8 M 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 47 M 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 264 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 748 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 220 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 27 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 18 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T08:26:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 31 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 745 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 127 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 162 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 320 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 298 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 99 k 2024-03-08T08:26:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 85 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 170 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T08:26:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.4 M 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T08:26:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 20 M 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 31 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 2.1 M 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 248 k 2024-03-08T08:26:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 4.7 M 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 17 M 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 15 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 25 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 163 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T08:26:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 501 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.1 M 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T08:26:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 45 k 2024-03-08T08:26:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 131 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T08:26:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T08:26:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T08:26:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T08:26:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-08T08:26:20.587 INFO:teuthology.orchestra.run.smithi012.stdout:Install 150 Packages 2024-03-08T08:26:20.588 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 24 Packages 2024-03-08T08:26:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:26:20.588 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 223 M 2024-03-08T08:26:20.588 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-08T08:26:21.353 INFO:teuthology.orchestra.run.smithi012.stdout:(1/174): ceph-19.0.0-1670.g1fb5d106.el9.x86_64. 48 kB/s | 6.6 kB 00:00 2024-03-08T08:26:21.536 INFO:teuthology.orchestra.run.smithi012.stdout:(2/174): ceph-fuse-19.0.0-1670.g1fb5d106.el9.x8 4.7 MB/s | 887 kB 00:00 2024-03-08T08:26:21.929 INFO:teuthology.orchestra.run.smithi012.stdout:(3/174): ceph-immutable-object-cache-19.0.0-167 371 kB/s | 145 kB 00:00 2024-03-08T08:26:22.312 INFO:teuthology.orchestra.run.smithi012.stdout:(4/174): ceph-mds-19.0.0-1670.g1fb5d106.el9.x86 5.6 MB/s | 2.1 MB 00:00 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T08:26:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T08:26:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T08:26:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T08:26:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T08:26:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T08:26:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T08:26:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T08:26:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T08:26:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T08:26:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T08:26:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T08:26:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T08:26:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T08:26:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T08:26:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:26:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T08:26:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T08:26:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T08:26:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T08:26:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T08:26:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T08:26:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T08:26:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:26:22.362 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-08T08:26:22.620 INFO:teuthology.orchestra.run.smithi012.stdout:(5/174): ceph-mgr-19.0.0-1670.g1fb5d106.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-03-08T08:26:22.866 DEBUG:teuthology.parallel:result is None 2024-03-08T08:26:23.079 INFO:teuthology.orchestra.run.smithi012.stdout:(6/174): ceph-common-19.0.0-1670.g1fb5d106.el9. 11 MB/s | 20 MB 00:01 2024-03-08T08:26:23.538 INFO:teuthology.orchestra.run.smithi012.stdout:(7/174): ceph-mon-19.0.0-1670.g1fb5d106.el9.x86 5.1 MB/s | 4.7 MB 00:00 2024-03-08T08:26:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:(8/174): ceph-osd-19.0.0-1670.g1fb5d106.el9.x86 14 MB/s | 17 MB 00:01 2024-03-08T08:26:24.296 INFO:teuthology.orchestra.run.smithi012.stdout:(9/174): ceph-selinux-19.0.0-1670.g1fb5d106.el9 609 kB/s | 25 kB 00:00 2024-03-08T08:26:24.871 INFO:teuthology.orchestra.run.smithi012.stdout:(10/174): ceph-radosgw-19.0.0-1670.g1fb5d106.el 7.3 MB/s | 9.8 MB 00:01 2024-03-08T08:26:24.938 INFO:teuthology.orchestra.run.smithi012.stdout:(11/174): cephfs-mirror-19.0.0-1670.g1fb5d106.e 3.2 MB/s | 220 kB 00:00 2024-03-08T08:26:24.997 INFO:teuthology.orchestra.run.smithi012.stdout:(12/174): cephfs-shell-19.0.0-1670.g1fb5d106.el 464 kB/s | 27 kB 00:00 2024-03-08T08:26:25.056 INFO:teuthology.orchestra.run.smithi012.stdout:(13/174): libcephfs-devel-19.0.0-1670.g1fb5d106 536 kB/s | 31 kB 00:00 2024-03-08T08:26:25.206 INFO:teuthology.orchestra.run.smithi012.stdout:(14/174): libcephfs2-19.0.0-1670.g1fb5d106.el9. 4.9 MB/s | 745 kB 00:00 2024-03-08T08:26:25.273 INFO:teuthology.orchestra.run.smithi012.stdout:(15/174): libcephsqlite-19.0.0-1670.g1fb5d106.e 2.4 MB/s | 163 kB 00:00 2024-03-08T08:26:25.381 INFO:teuthology.orchestra.run.smithi012.stdout:(16/174): librados-devel-19.0.0-1670.g1fb5d106. 1.1 MB/s | 127 kB 00:00 2024-03-08T08:26:25.498 INFO:teuthology.orchestra.run.smithi012.stdout:(17/174): libradosstriper1-19.0.0-1670.g1fb5d10 4.2 MB/s | 501 kB 00:00 2024-03-08T08:26:26.224 INFO:teuthology.orchestra.run.smithi012.stdout:(18/174): librgw2-19.0.0-1670.g1fb5d106.el9.x86 7.1 MB/s | 5.1 MB 00:00 2024-03-08T08:26:26.265 INFO:teuthology.orchestra.run.smithi012.stdout:(19/174): python3-ceph-argparse-19.0.0-1670.g1f 1.1 MB/s | 45 kB 00:00 2024-03-08T08:26:26.316 INFO:teuthology.orchestra.run.smithi012.stdout:(20/174): python3-ceph-common-19.0.0-1670.g1fb5 2.6 MB/s | 131 kB 00:00 2024-03-08T08:26:26.383 INFO:teuthology.orchestra.run.smithi012.stdout:(21/174): python3-cephfs-19.0.0-1670.g1fb5d106. 2.4 MB/s | 162 kB 00:00 2024-03-08T08:26:26.466 INFO:teuthology.orchestra.run.smithi012.stdout:(22/174): python3-rados-19.0.0-1670.g1fb5d106.e 3.8 MB/s | 320 kB 00:00 2024-03-08T08:26:26.542 INFO:teuthology.orchestra.run.smithi012.stdout:(23/174): python3-rbd-19.0.0-1670.g1fb5d106.el9 3.9 MB/s | 298 kB 00:00 2024-03-08T08:26:26.600 INFO:teuthology.orchestra.run.smithi012.stdout:(24/174): python3-rgw-19.0.0-1670.g1fb5d106.el9 1.7 MB/s | 99 kB 00:00 2024-03-08T08:26:26.667 INFO:teuthology.orchestra.run.smithi012.stdout:(25/174): rbd-fuse-19.0.0-1670.g1fb5d106.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-08T08:26:27.101 INFO:teuthology.orchestra.run.smithi012.stdout:(26/174): ceph-test-19.0.0-1670.g1fb5d106.el9.x 17 MB/s | 47 MB 00:02 2024-03-08T08:26:27.168 INFO:teuthology.orchestra.run.smithi012.stdout:(27/174): ceph-base-19.0.0-1670.g1fb5d106.el9.x 920 kB/s | 5.3 MB 00:05 2024-03-08T08:26:27.193 INFO:teuthology.orchestra.run.smithi012.stdout:(28/174): rbd-nbd-19.0.0-1670.g1fb5d106.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-08T08:26:27.260 INFO:teuthology.orchestra.run.smithi012.stdout:(29/174): rbd-mirror-19.0.0-1670.g1fb5d106.el9. 5.2 MB/s | 3.1 MB 00:00 2024-03-08T08:26:27.277 INFO:teuthology.orchestra.run.smithi012.stdout:(30/174): ceph-grafana-dashboards-19.0.0-1670.g 288 kB/s | 31 kB 00:00 2024-03-08T08:26:27.302 INFO:teuthology.orchestra.run.smithi012.stdout:(31/174): ceph-mgr-cephadm-19.0.0-1670.g1fb5d10 1.2 MB/s | 138 kB 00:00 2024-03-08T08:26:27.361 INFO:teuthology.orchestra.run.smithi012.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1670.g1f 4.2 MB/s | 248 kB 00:00 2024-03-08T08:26:27.411 INFO:teuthology.orchestra.run.smithi012.stdout:(33/174): ceph-mgr-rook-19.0.0-1670.g1fb5d106.e 991 kB/s | 49 kB 00:00 2024-03-08T08:26:27.470 INFO:teuthology.orchestra.run.smithi012.stdout:(34/174): ceph-mgr-dashboard-19.0.0-1670.g1fb5d 8.5 MB/s | 1.8 MB 00:00 2024-03-08T08:26:27.487 INFO:teuthology.orchestra.run.smithi012.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1670.g1 204 kB/s | 15 kB 00:00 2024-03-08T08:26:27.520 INFO:teuthology.orchestra.run.smithi012.stdout:(36/174): ceph-volume-19.0.0-1670.g1fb5d106.el9 5.1 MB/s | 264 kB 00:00 2024-03-08T08:26:27.579 INFO:teuthology.orchestra.run.smithi012.stdout:(37/174): cephadm-19.0.0-1670.g1fb5d106.el9.noa 7.9 MB/s | 748 kB 00:00 2024-03-08T08:26:27.596 INFO:teuthology.orchestra.run.smithi012.stdout:(38/174): cephfs-top-19.0.0-1670.g1fb5d106.el9. 243 kB/s | 18 kB 00:00 2024-03-08T08:26:28.071 INFO:teuthology.orchestra.run.smithi012.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 662 kB 00:00 2024-03-08T08:26:28.205 INFO:teuthology.orchestra.run.smithi012.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 920 kB 00:00 2024-03-08T08:26:28.222 INFO:teuthology.orchestra.run.smithi012.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 706 kB/s | 106 kB 00:00 2024-03-08T08:26:28.372 INFO:teuthology.orchestra.run.smithi012.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 307 kB 00:00 2024-03-08T08:26:28.414 INFO:teuthology.orchestra.run.smithi012.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 165 kB/s | 32 kB 00:00 2024-03-08T08:26:28.514 INFO:teuthology.orchestra.run.smithi012.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 172 kB/s | 17 kB 00:00 2024-03-08T08:26:28.565 INFO:teuthology.orchestra.run.smithi012.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-08T08:26:28.582 INFO:teuthology.orchestra.run.smithi012.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 154 kB/s | 10 kB 00:00 2024-03-08T08:26:28.607 INFO:teuthology.orchestra.run.smithi012.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-08T08:26:28.624 INFO:teuthology.orchestra.run.smithi012.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-08T08:26:28.674 INFO:teuthology.orchestra.run.smithi012.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-08T08:26:28.708 INFO:teuthology.orchestra.run.smithi012.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-08T08:26:28.725 INFO:teuthology.orchestra.run.smithi012.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-08T08:26:28.758 INFO:teuthology.orchestra.run.smithi012.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-08T08:26:28.783 INFO:teuthology.orchestra.run.smithi012.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-08T08:26:28.809 INFO:teuthology.orchestra.run.smithi012.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-08T08:26:28.834 INFO:teuthology.orchestra.run.smithi012.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-08T08:26:28.976 INFO:teuthology.orchestra.run.smithi012.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-08T08:26:29.018 INFO:teuthology.orchestra.run.smithi012.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.3 MB/s | 1.5 MB 00:00 2024-03-08T08:26:29.043 INFO:teuthology.orchestra.run.smithi012.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-08T08:26:29.069 INFO:teuthology.orchestra.run.smithi012.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-08T08:26:29.253 INFO:teuthology.orchestra.run.smithi012.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-08T08:26:29.286 INFO:teuthology.orchestra.run.smithi012.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-03-08T08:26:29.328 INFO:teuthology.orchestra.run.smithi012.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-08T08:26:29.512 INFO:teuthology.orchestra.run.smithi012.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-08T08:26:29.528 INFO:teuthology.orchestra.run.smithi012.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 123 kB/s | 24 kB 00:00 2024-03-08T08:26:29.570 INFO:teuthology.orchestra.run.smithi012.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 436 kB/s | 18 kB 00:00 2024-03-08T08:26:29.595 INFO:teuthology.orchestra.run.smithi012.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-08T08:26:29.612 INFO:teuthology.orchestra.run.smithi012.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 754 kB/s | 32 kB 00:00 2024-03-08T08:26:29.838 INFO:teuthology.orchestra.run.smithi012.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-08T08:26:29.863 INFO:teuthology.orchestra.run.smithi012.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 868 kB/s | 217 kB 00:00 2024-03-08T08:26:29.896 INFO:teuthology.orchestra.run.smithi012.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-08T08:26:29.922 INFO:teuthology.orchestra.run.smithi012.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-08T08:26:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 982 kB/s | 41 kB 00:00 2024-03-08T08:26:29.981 INFO:teuthology.orchestra.run.smithi012.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-08T08:26:30.006 INFO:teuthology.orchestra.run.smithi012.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-08T08:26:30.040 INFO:teuthology.orchestra.run.smithi012.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-08T08:26:30.056 INFO:teuthology.orchestra.run.smithi012.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-03-08T08:26:30.082 INFO:teuthology.orchestra.run.smithi012.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-08T08:26:30.699 INFO:teuthology.orchestra.run.smithi012.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 63 kB/s | 39 kB 00:00 2024-03-08T08:26:30.858 INFO:teuthology.orchestra.run.smithi012.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-08T08:26:30.874 INFO:teuthology.orchestra.run.smithi012.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 235 kB/s | 41 kB 00:00 2024-03-08T08:26:30.991 INFO:teuthology.orchestra.run.smithi012.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 300 kB 00:00 2024-03-08T08:26:31.008 INFO:teuthology.orchestra.run.smithi012.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 423 kB/s | 56 kB 00:00 2024-03-08T08:26:31.058 INFO:teuthology.orchestra.run.smithi012.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-08T08:26:31.075 INFO:teuthology.orchestra.run.smithi012.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 659 kB/s | 44 kB 00:00 2024-03-08T08:26:31.117 INFO:teuthology.orchestra.run.smithi012.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-08T08:26:31.143 INFO:teuthology.orchestra.run.smithi012.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-08T08:26:31.235 INFO:teuthology.orchestra.run.smithi012.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.1 MB 00:00 2024-03-08T08:26:31.260 INFO:teuthology.orchestra.run.smithi012.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 881 kB/s | 103 kB 00:00 2024-03-08T08:26:31.294 INFO:teuthology.orchestra.run.smithi012.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-08T08:26:31.327 INFO:teuthology.orchestra.run.smithi012.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-08T08:26:31.586 INFO:teuthology.orchestra.run.smithi012.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 641 kB/s | 187 kB 00:00 2024-03-08T08:26:31.719 INFO:teuthology.orchestra.run.smithi012.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-08T08:26:31.812 INFO:teuthology.orchestra.run.smithi012.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-08T08:26:31.837 INFO:teuthology.orchestra.run.smithi012.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-08T08:26:32.012 INFO:teuthology.orchestra.run.smithi012.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 107 kB/s | 21 kB 00:00 2024-03-08T08:26:32.046 INFO:teuthology.orchestra.run.smithi012.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 122 kB/s | 25 kB 00:00 2024-03-08T08:26:32.271 INFO:teuthology.orchestra.run.smithi012.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-08T08:26:32.363 INFO:teuthology.orchestra.run.smithi012.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-08T08:26:32.446 INFO:teuthology.orchestra.run.smithi012.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-08T08:26:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.2 MB/s | 4.1 MB 00:00 2024-03-08T08:26:32.538 INFO:teuthology.orchestra.run.smithi012.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 379 kB/s | 35 kB 00:00 2024-03-08T08:26:32.581 INFO:teuthology.orchestra.run.smithi012.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 667 kB/s | 44 kB 00:00 2024-03-08T08:26:32.614 INFO:teuthology.orchestra.run.smithi012.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-08T08:26:39.072 INFO:teuthology.orchestra.run.smithi012.stdout:(104/174): ceph-mgr-diskprediction-local-19.0.0 642 kB/s | 7.4 MB 00:11 2024-03-08T08:26:40.830 INFO:teuthology.orchestra.run.smithi012.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 312 kB/s | 548 kB 00:01 2024-03-08T08:26:41.855 INFO:teuthology.orchestra.run.smithi012.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 301 kB/s | 309 kB 00:01 2024-03-08T08:26:41.939 INFO:teuthology.orchestra.run.smithi012.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-08T08:26:51.812 INFO:teuthology.orchestra.run.smithi012.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 458 kB/s | 4.4 MB 00:09 2024-03-08T08:26:51.904 INFO:teuthology.orchestra.run.smithi012.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-08T08:26:52.171 INFO:teuthology.orchestra.run.smithi012.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 182 kB/s | 49 kB 00:00 2024-03-08T08:26:52.529 INFO:teuthology.orchestra.run.smithi012.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-03-08T08:26:53.154 INFO:teuthology.orchestra.run.smithi012.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 242 kB/s | 151 kB 00:00 2024-03-08T08:26:54.530 INFO:teuthology.orchestra.run.smithi012.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 609 kB/s | 838 kB 00:01 2024-03-08T08:26:55.355 INFO:teuthology.orchestra.run.smithi012.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 664 kB/s | 548 kB 00:00 2024-03-08T08:26:55.472 INFO:teuthology.orchestra.run.smithi012.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 372 kB/s | 43 kB 00:00 2024-03-08T08:26:55.530 INFO:teuthology.orchestra.run.smithi012.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 553 kB/s | 32 kB 00:00 2024-03-08T08:26:55.589 INFO:teuthology.orchestra.run.smithi012.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-08T08:26:55.856 INFO:teuthology.orchestra.run.smithi012.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-03-08T08:26:56.331 INFO:teuthology.orchestra.run.smithi012.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 755 kB/s | 358 kB 00:00 2024-03-08T08:26:56.781 INFO:teuthology.orchestra.run.smithi012.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 674 kB/s | 303 kB 00:00 2024-03-08T08:26:56.840 INFO:teuthology.orchestra.run.smithi012.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 607 kB/s | 35 kB 00:00 2024-03-08T08:26:57.123 INFO:teuthology.orchestra.run.smithi012.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 715 kB/s | 203 kB 00:00 2024-03-08T08:26:59.181 INFO:teuthology.orchestra.run.smithi012.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.0 MB/s | 2.0 MB 00:02 2024-03-08T08:26:59.357 INFO:teuthology.orchestra.run.smithi012.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 826 kB/s | 144 kB 00:00 2024-03-08T08:26:59.432 INFO:teuthology.orchestra.run.smithi012.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-08T08:26:59.516 INFO:teuthology.orchestra.run.smithi012.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-03-08T08:26:59.599 INFO:teuthology.orchestra.run.smithi012.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 279 kB/s | 23 kB 00:00 2024-03-08T08:26:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-08T08:27:00.108 INFO:teuthology.orchestra.run.smithi012.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 45 kB/s | 20 kB 00:00 2024-03-08T08:27:02.124 INFO:teuthology.orchestra.run.smithi012.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 518 kB/s | 1.0 MB 00:02 2024-03-08T08:27:02.241 INFO:teuthology.orchestra.run.smithi012.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 397 kB/s | 46 kB 00:00 2024-03-08T08:27:02.408 INFO:teuthology.orchestra.run.smithi012.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 473 kB/s | 79 kB 00:00 2024-03-08T08:27:02.517 INFO:teuthology.orchestra.run.smithi012.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-03-08T08:27:03.050 INFO:teuthology.orchestra.run.smithi012.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 511 kB/s | 272 kB 00:00 2024-03-08T08:27:03.117 INFO:teuthology.orchestra.run.smithi012.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-08T08:27:03.384 INFO:teuthology.orchestra.run.smithi012.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 337 kB/s | 90 kB 00:00 2024-03-08T08:27:03.518 INFO:teuthology.orchestra.run.smithi012.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 212 kB/s | 28 kB 00:00 2024-03-08T08:27:03.626 INFO:teuthology.orchestra.run.smithi012.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-03-08T08:27:04.076 INFO:teuthology.orchestra.run.smithi012.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 419 kB/s | 188 kB 00:00 2024-03-08T08:27:04.277 INFO:teuthology.orchestra.run.smithi012.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-03-08T08:27:04.402 INFO:teuthology.orchestra.run.smithi012.stdout:(141/174): lua-devel-5.4.4-4.el9.x86_64.rpm 716 B/s | 22 kB 00:31 2024-03-08T08:27:04.419 INFO:teuthology.orchestra.run.smithi012.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-03-08T08:27:04.519 INFO:teuthology.orchestra.run.smithi012.stdout:(143/174): protobuf-compiler-3.14.0-13.el9.x86_ 27 kB/s | 863 kB 00:31 2024-03-08T08:27:04.603 INFO:teuthology.orchestra.run.smithi012.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 254 kB/s | 51 kB 00:00 2024-03-08T08:27:04.895 INFO:teuthology.orchestra.run.smithi012.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 239 kB/s | 90 kB 00:00 2024-03-08T08:27:04.995 INFO:teuthology.orchestra.run.smithi012.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-08T08:27:05.104 INFO:teuthology.orchestra.run.smithi012.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 335 kB/s | 230 kB 00:00 2024-03-08T08:27:05.412 INFO:teuthology.orchestra.run.smithi012.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 460 kB/s | 192 kB 00:00 2024-03-08T08:27:05.538 INFO:teuthology.orchestra.run.smithi012.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 457 kB/s | 427 kB 00:00 2024-03-08T08:27:05.771 INFO:teuthology.orchestra.run.smithi012.stdout:(150/174): librados2-19.0.0-1670.g1fb5d106.el9. 9.4 MB/s | 3.4 MB 00:00 2024-03-08T08:27:05.854 INFO:teuthology.orchestra.run.smithi012.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2024-03-08T08:27:06.896 INFO:teuthology.orchestra.run.smithi012.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-03-08T08:27:07.705 INFO:teuthology.orchestra.run.smithi012.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.2 MB/s | 1.8 MB 00:00 2024-03-08T08:27:07.788 INFO:teuthology.orchestra.run.smithi012.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 509 kB/s | 42 kB 00:00 2024-03-08T08:27:07.939 INFO:teuthology.orchestra.run.smithi012.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.6 MB/s | 248 kB 00:00 2024-03-08T08:27:08.014 INFO:teuthology.orchestra.run.smithi012.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 953 kB/s | 71 kB 00:00 2024-03-08T08:27:08.097 INFO:teuthology.orchestra.run.smithi012.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-08T08:27:08.306 INFO:teuthology.orchestra.run.smithi012.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 923 kB/s | 192 kB 00:00 2024-03-08T08:27:08.398 INFO:teuthology.orchestra.run.smithi012.stdout:(159/174): thrift-0.15.0-2.el9.x86_64.rpm 492 kB/s | 1.6 MB 00:03 2024-03-08T08:27:08.548 INFO:teuthology.orchestra.run.smithi012.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.0 MB/s | 256 kB 00:00 2024-03-08T08:27:08.573 INFO:teuthology.orchestra.run.smithi012.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 131 kB/s | 23 kB 00:00 2024-03-08T08:27:08.641 INFO:teuthology.orchestra.run.smithi012.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-08T08:27:08.791 INFO:teuthology.orchestra.run.smithi012.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 905 kB/s | 136 kB 00:00 2024-03-08T08:27:08.874 INFO:teuthology.orchestra.run.smithi012.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 767 kB/s | 64 kB 00:00 2024-03-08T08:27:08.958 INFO:teuthology.orchestra.run.smithi012.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 345 kB/s | 29 kB 00:00 2024-03-08T08:27:09.408 INFO:teuthology.orchestra.run.smithi012.stdout:(166/174): librbd1-19.0.0-1670.g1fb5d106.el9.x8 829 kB/s | 3.1 MB 00:03 2024-03-08T08:27:09.584 INFO:teuthology.orchestra.run.smithi012.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 1.9 MB/s | 1.2 MB 00:00 2024-03-08T08:27:09.859 INFO:teuthology.orchestra.run.smithi012.stdout:(168/174): libfdisk-2.37.4-18.el9.x86_64.rpm 120 kB/s | 155 kB 00:01 2024-03-08T08:27:10.584 INFO:teuthology.orchestra.run.smithi012.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.8 MB/s | 2.2 MB 00:01 2024-03-08T08:27:10.659 INFO:teuthology.orchestra.run.smithi012.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 416 kB/s | 31 kB 00:00 2024-03-08T08:27:10.826 INFO:teuthology.orchestra.run.smithi012.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 2.3 MB 00:01 2024-03-08T08:27:11.035 INFO:teuthology.orchestra.run.smithi012.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 357 kB/s | 74 kB 00:00 2024-03-08T08:27:11.060 INFO:teuthology.orchestra.run.smithi012.stdout:(173/174): util-linux-core-2.37.4-18.el9.x86_64 387 kB/s | 465 kB 00:01 2024-03-08T08:27:11.219 INFO:teuthology.orchestra.run.smithi012.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 1.9 MB/s | 1.1 MB 00:00 2024-03-08T08:27:11.226 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-08T08:27:11.226 INFO:teuthology.orchestra.run.smithi012.stdout:Total 4.4 MB/s | 223 MB 00:50 2024-03-08T08:27:11.685 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-08T08:27:12.016 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-08T08:27:12.016 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-08T08:27:14.099 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-08T08:27:14.099 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-08T08:27:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-08T08:27:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T08:27:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T08:27:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T08:27:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T08:27:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T08:27:21.466 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T08:27:21.499 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T08:27:21.532 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T08:27:21.581 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T08:27:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T08:27:21.985 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:27:22.094 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:27:22.128 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:27:22.204 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:27:22.238 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/198 2024-03-08T08:27:22.285 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T08:27:22.475 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T08:27:22.531 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T08:27:22.558 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:27:22.646 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:27:22.676 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 17/198 2024-03-08T08:27:22.724 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 17/198 2024-03-08T08:27:22.763 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T08:27:22.786 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T08:27:22.801 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T08:27:22.998 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T08:27:23.034 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T08:27:23.066 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T08:27:23.246 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T08:27:23.396 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T08:27:23.457 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T08:27:23.551 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:27:23.599 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:27:23.814 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T08:27:23.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T08:27:24.019 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T08:27:24.071 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T08:27:24.098 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T08:27:24.142 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T08:27:24.218 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T08:27:24.274 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T08:27:24.311 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T08:27:24.377 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T08:27:24.418 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T08:27:24.463 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T08:27:24.495 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T08:27:24.522 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T08:27:24.594 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T08:27:24.627 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9. 39/198 2024-03-08T08:27:24.680 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el 40/198 2024-03-08T08:27:24.707 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_6 41/198 2024-03-08T08:27:24.780 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T08:27:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T08:27:24.906 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T08:27:25.059 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T08:27:25.108 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T08:27:25.659 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T08:27:25.738 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T08:27:25.828 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T08:27:25.889 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T08:27:25.897 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T08:27:25.938 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T08:27:25.987 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T08:27:26.025 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_6 53/198 2024-03-08T08:27:26.055 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T08:27:26.331 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T08:27:26.486 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T08:27:26.574 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T08:27:26.618 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T08:27:26.759 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T08:27:26.820 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T08:27:26.876 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T08:27:26.951 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T08:27:26.987 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T08:27:27.024 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T08:27:27.071 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T08:27:27.107 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T08:27:27.150 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T08:27:27.192 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T08:27:27.332 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T08:27:27.380 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T08:27:27.411 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T08:27:27.453 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T08:27:27.549 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T08:27:27.585 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T08:27:27.617 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T08:27:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T08:27:27.667 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T08:27:27.754 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T08:27:27.889 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T08:27:27.995 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T08:27:28.034 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T08:27:28.083 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T08:27:28.139 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T08:27:28.329 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T08:27:28.371 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T08:27:28.513 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T08:27:28.553 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T08:27:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T08:27:28.624 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T08:27:28.652 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T08:27:28.686 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T08:27:28.725 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T08:27:28.757 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T08:27:28.786 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T08:27:28.865 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T08:27:28.897 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T08:27:28.930 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T08:27:28.972 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T08:27:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T08:27:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T08:27:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T08:27:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T08:27:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T08:27:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T08:27:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T08:27:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T08:27:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T08:27:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T08:27:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T08:27:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T08:27:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T08:27:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T08:27:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T08:27:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T08:27:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T08:27:31.454 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T08:27:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T08:27:32.869 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T08:27:32.974 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T08:27:33.005 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T08:27:33.036 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T08:27:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T08:27:33.388 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T08:27:33.419 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 124/198 2024-03-08T08:27:33.493 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 124/198 2024-03-08T08:27:33.523 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 125/198 2024-03-08T08:27:33.560 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T08:27:34.037 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T08:27:34.063 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 128/198 2024-03-08T08:27:34.117 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 128/198 2024-03-08T08:27:34.139 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 129/198 2024-03-08T08:27:36.439 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:27:36.477 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:27:37.043 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 130/198 2024-03-08T08:27:37.080 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 131/198 2024-03-08T08:27:37.166 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 131/198 2024-03-08T08:27:37.166 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T08:27:37.166 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T08:27:37.166 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:37.198 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 132/198 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 132/198 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-03-08T08:27:49.512 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:49.677 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 133/198 2024-03-08T08:27:49.691 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 133/198 2024-03-08T08:27:49.692 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:49.692 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T08:27:49.692 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:51.221 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 134/198 2024-03-08T08:27:51.246 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 134/198 2024-03-08T08:27:51.246 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:51.246 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T08:27:51.246 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T08:27:51.246 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T08:27:51.247 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:51.455 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 135/198 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 135/198 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T08:27:51.477 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:51.939 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 136/198 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 136/198 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T08:27:51.961 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:52.058 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T08:27:52.417 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 138/198 2024-03-08T08:27:52.448 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 138/198 2024-03-08T08:27:52.480 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.e 139/198 2024-03-08T08:27:52.542 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106. 140/198 2024-03-08T08:27:52.571 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 141/198 2024-03-08T08:27:52.810 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 141/198 2024-03-08T08:27:52.846 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 142/198 2024-03-08T08:27:53.858 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 142/198 2024-03-08T08:27:53.888 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 143/198 2024-03-08T08:27:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 143/198 2024-03-08T08:27:54.159 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el 144/198 2024-03-08T08:27:54.190 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 145/198 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 145/198 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T08:27:54.213 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:54.258 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 146/198 2024-03-08T08:27:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 146/198 2024-03-08T08:27:54.325 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 147/198 2024-03-08T08:27:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T08:27:55.252 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 149/198 2024-03-08T08:27:55.274 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 149/198 2024-03-08T08:27:55.274 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:55.274 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T08:27:55.274 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T08:27:55.274 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T08:27:55.275 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:55.310 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 150/198 2024-03-08T08:27:55.332 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 150/198 2024-03-08T08:27:55.332 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:55.332 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T08:27:55.332 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:55.378 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:27:55.399 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:27:55.399 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:55.399 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T08:27:55.399 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T08:27:55.399 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T08:27:55.400 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:27:55.737 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 152/198 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 152/198 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T08:27:55.759 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:00.441 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 153/198 2024-03-08T08:28:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 154/198 2024-03-08T08:28:00.512 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 155/198 2024-03-08T08:28:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T08:28:00.737 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 157/198 2024-03-08T08:28:00.859 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T08:28:00.904 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 159/198 2024-03-08T08:28:00.965 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T08:28:01.021 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T08:28:01.064 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T08:28:01.113 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_ 163/198 2024-03-08T08:28:01.154 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T08:28:01.184 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T08:28:01.246 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T08:28:01.313 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T08:28:01.340 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T08:28:01.928 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:28:01.962 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:28:01.982 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T08:28:02.854 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 170/198 2024-03-08T08:28:02.934 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T08:28:02.976 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T08:28:03.076 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T08:28:03.113 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T08:28:03.138 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T08:28:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T08:28:03.198 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T08:28:03.198 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T08:28:03.278 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T08:28:03.307 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T08:28:03.307 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T08:28:03.384 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T08:28:03.410 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T08:28:03.443 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T08:28:03.468 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T08:28:03.493 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T08:28:03.529 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T08:28:03.574 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T08:28:03.606 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T08:28:03.631 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T08:28:03.657 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T08:28:03.682 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T08:28:03.707 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T08:28:03.713 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:28:03.714 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:28:03.901 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T08:28:03.902 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T08:28:03.976 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T08:28:04.001 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T08:28:04.026 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T08:28:04.059 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T08:28:04.085 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T08:28:04.154 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T08:28:08.837 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T08:28:08.837 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/198 2024-03-08T08:28:08.837 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/198 2024-03-08T08:28:08.837 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/198 2024-03-08T08:28:08.837 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d 5/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 8/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 13/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 14/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_ 15/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 16/198 2024-03-08T08:28:08.838 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 17/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_6 18/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86 19/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 20/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el 21/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9. 22/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_6 23/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 24/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 25/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/198 2024-03-08T08:28:08.839 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 27/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 28/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106. 30/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noa 31/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.n 32/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb 33/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el 34/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 35/198 2024-03-08T08:28:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.e 36/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 37/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 38/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 39/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T08:28:08.856 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T08:28:08.857 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T08:28:08.858 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T08:28:08.859 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T08:28:08.860 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T08:28:08.861 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T08:28:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T08:28:08.863 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 151/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 153/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T08:28:08.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T08:28:08.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T08:28:08.866 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T08:28:08.867 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T08:28:11.696 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.697 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T08:28:11.698 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T08:28:11.700 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.701 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T08:28:11.702 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T08:28:11.703 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T08:28:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T08:28:11.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T08:28:11.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T08:28:11.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T08:28:11.708 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T08:28:11.709 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T08:28:11.710 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T08:28:11.710 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:11.710 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-08T08:28:12.137 DEBUG:teuthology.parallel:result is None 2024-03-08T08:28:12.138 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-08T08:28:12.496 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T08:28:12.518 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-1670.g1fb5d106.el9 2024-03-08T08:28:12.519 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1670.g1fb5d106.el9 2024-03-08T08:28:12.519 INFO:teuthology.task.install:The correct ceph version 19.0.0-1670.g1fb5d106 is installed. 2024-03-08T08:28:12.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-08T08:28:12.892 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T08:28:12.917 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-1670.g1fb5d106.el9 2024-03-08T08:28:12.918 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1670.g1fb5d106.el9 2024-03-08T08:28:12.918 INFO:teuthology.task.install:The correct ceph version 19.0.0-1670.g1fb5d106 is installed. 2024-03-08T08:28:12.920 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-08T08:28:12.920 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:12.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T08:28:12.954 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:12.955 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T08:28:12.991 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-08T08:28:12.991 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:12.992 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T08:28:13.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T08:28:13.098 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:13.098 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T08:28:13.128 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T08:28:13.199 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-08T08:28:13.199 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:13.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T08:28:13.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T08:28:13.300 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:13.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T08:28:13.328 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T08:28:13.394 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-08T08:28:13.394 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:13.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T08:28:13.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T08:28:13.497 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:13.497 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T08:28:13.523 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T08:28:13.591 INFO:teuthology.run_tasks:Running task ceph... 2024-03-08T08:28:13.668 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-08T08:28:13.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-03-08T08:28:13.671 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2024-03-08T08:28:13.699 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-08T08:28:13.699 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-08T08:28:13.741 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-08T08:28:13.769 INFO:tasks.ceph:Creating extra log directories... 2024-03-08T08:28:13.769 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-08T08:28:13.811 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-08T08:28:13.850 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-08T08:28:13.850 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-08T08:28:13.850 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-08T08:28:13.850 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:13.850 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-08T08:28:13.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:14.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:14.045 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:14.045 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-08T08:28:14.060 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:14.177 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:14.208 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-08T08:28:14.208 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'missing required features'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-08T08:28:14.209 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510', 'branch': 'main', 'description': 'fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}', 'email': 'jcollin@redhat.com', 'first_in_suite': False, 'job_id': '7585510', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'missing required features'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-jcollin-test_per-client-perf-counters', 'sha1': 'f6f80007dcf9e039bb253dd50a92ba3c71c6943a'}}, 'owner': 'scheduled_jcollin@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7204, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'sleep_before_teardown': 0, 'suite': 'fs:functional', 'suite_branch': 'wip-jcollin-test_per-client-perf-counters', 'suite_path': '/home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/joscollin/ceph.git', 'suite_sha1': 'f6f80007dcf9e039bb253dd50a92ba3c71c6943a', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tSPXjRcxJrQoUq/rh0IHWWR+SpHIJgvsXRqThVIwRMImZoQVE9S+8dsvivc5X9Jd/QydIsWWroMlepiN1lIw=', 'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/z/uV83Z6d9tF1DAevK6aqUuf/yu+zfZfYdqeRUiZMOeCd3jl6UOBBo2DMw8+NT1UTHnp3ND54on+Yljk6URQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'missing required features'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_admin.TestLabeledPerfCounters']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-08_08:09:40', 'tube': 'smithi', 'user': 'jcollin', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-08T08:28:14.209 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-08T08:28:14.211 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-08T08:28:14.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-03-08T08:28:14.267 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2024-03-08T08:28:14.324 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:14.324 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T08:28:14.350 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T08:28:14.350 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-03-08T08:28:14.405 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-08 08:28:04.494152695 +0000 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-08 08:23:28.327211562 +0000 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-08 08:23:28.327211562 +0000 2024-03-08T08:28:14.406 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-08 08:23:28.327211562 +0000 2024-03-08T08:28:14.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T08:28:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-08T08:28:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-08T08:28:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000266952 s, 1.9 MB/s 2024-03-08T08:28:14.475 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T08:28:14.531 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-03-08T08:28:14.587 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T08:28:14.587 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:14.587 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-08T08:28:14.587 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:14.588 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:14.588 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-08 08:28:04.495152687 +0000 2024-03-08T08:28:14.588 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-08 08:23:28.668208588 +0000 2024-03-08T08:28:14.588 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-08 08:23:28.668208588 +0000 2024-03-08T08:28:14.588 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-08 08:23:28.668208588 +0000 2024-03-08T08:28:14.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T08:28:14.655 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-08T08:28:14.655 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-08T08:28:14.655 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000225924 s, 2.3 MB/s 2024-03-08T08:28:14.656 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T08:28:14.713 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-03-08T08:28:14.769 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-08 08:28:04.495152687 +0000 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-08 08:23:29.016205554 +0000 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-08 08:23:29.016205554 +0000 2024-03-08T08:28:14.770 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-08 08:23:29.016205554 +0000 2024-03-08T08:28:14.770 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T08:28:14.837 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-08T08:28:14.837 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-08T08:28:14.837 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000196459 s, 2.6 MB/s 2024-03-08T08:28:14.839 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T08:28:14.895 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-08 08:28:04.496152678 +0000 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-08 08:23:29.359202562 +0000 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-08 08:23:29.359202562 +0000 2024-03-08T08:28:14.952 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-08 08:23:29.359202562 +0000 2024-03-08T08:28:14.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T08:28:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-08T08:28:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-08T08:28:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000231162 s, 2.2 MB/s 2024-03-08T08:28:15.020 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T08:28:15.077 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:15.077 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:15.077 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T08:28:15.094 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T08:28:15.094 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:26:14.748009624 +0000 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:23:28.250033526 +0000 2024-03-08T08:28:15.149 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:23:28.250033526 +0000 2024-03-08T08:28:15.150 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-08 08:23:28.250033526 +0000 2024-03-08T08:28:15.150 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T08:28:15.215 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-08T08:28:15.216 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-08T08:28:15.216 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000138633 s, 3.7 MB/s 2024-03-08T08:28:15.217 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T08:28:15.274 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-03-08T08:28:15.329 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T08:28:15.329 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:26:14.749009609 +0000 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:23:28.607028044 +0000 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:23:28.607028044 +0000 2024-03-08T08:28:15.330 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-08 08:23:28.607028044 +0000 2024-03-08T08:28:15.330 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T08:28:15.395 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-08T08:28:15.395 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-08T08:28:15.395 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000105938 s, 4.8 MB/s 2024-03-08T08:28:15.396 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T08:28:15.452 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:26:14.749009609 +0000 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:23:28.942022898 +0000 2024-03-08T08:28:15.507 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:23:28.942022898 +0000 2024-03-08T08:28:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-08 08:23:28.942022898 +0000 2024-03-08T08:28:15.508 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T08:28:15.573 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-08T08:28:15.573 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-08T08:28:15.573 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 7.9829e-05 s, 6.4 MB/s 2024-03-08T08:28:15.574 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T08:28:15.629 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:26:14.749009609 +0000 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:23:29.278017738 +0000 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:23:29.279017723 +0000 2024-03-08T08:28:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-08 08:23:29.278017738 +0000 2024-03-08T08:28:15.684 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T08:28:15.749 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-08T08:28:15.749 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-08T08:28:15.749 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000101653 s, 5.0 MB/s 2024-03-08T08:28:15.750 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T08:28:15.806 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:15.806 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-08T08:28:15.806 INFO:tasks.ceph:Generating config... 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] debug client = 20 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] debug ms = 1 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] pid file = 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-08T08:28:15.808 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-08T08:28:15.808 INFO:tasks.ceph:[global] lockdep = True 2024-03-08T08:28:15.808 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-08T08:28:15.808 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-08T08:28:15.809 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-08T08:28:15.810 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-08T08:28:15.810 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-08T08:28:15.811 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-08T08:28:15.811 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-08T08:28:15.811 INFO:tasks.ceph:Setting up mon.a... 2024-03-08T08:28:15.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-08T08:28:15.868 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2024-03-08T08:28:15.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-08T08:28:15.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-08T08:28:15.944 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.b', '172.21.15.156'), ('mon.c', '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]')] 2024-03-08T08:28:15.944 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]', 'lockdep': True}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-08T08:28:15.945 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:15.945 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-08T08:28:15.999 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --add b 172.21.15.156 --addv c '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:fsid e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2024-03-08T08:28:16.085836+0000 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-08T08:28:16.085836+0000 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 17 (quincy) 2024-03-08T08:28:16.086 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2024-03-08T08:28:16.087 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-08T08:28:16.087 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-08T08:28:16.087 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] mon.c 2024-03-08T08:28:16.087 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-08T08:28:16.088 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-08T08:28:16.102 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c... 2024-03-08T08:28:16.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-08T08:28:16.144 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-03-08T08:28:16.192 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-03-08T08:28:16.193 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.194 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: lockdep = True 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-03-08T08:28:16.195 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.196 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bdev async discard = True 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bdev enable discard = True 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore allocator = bitmap 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 1/20 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 1/20 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 4/10 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2024-03-08T08:28:16.197 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-03-08T08:28:16.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-08T08:28:16.199 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: mon op complaint time = 120 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: client mount timeout = 600 2024-03-08T08:28:16.200 INFO:teuthology.orchestra.run.smithi012.stdout: debug client = 20 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = "" 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout:[mds] 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds = 20 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds balancer = 20 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-08T08:28:16.201 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug frag = True 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug scatterstat = True 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: mds op complaint time = 180 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: mds verify scatter = True 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-08T08:28:16.202 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-08T08:28:16.203 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-08T08:28:16.204 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.12,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: lockdep = True 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2024-03-08T08:28:16.205 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug trim objects = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.206 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bdev async discard = True 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bdev enable discard = True 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore allocator = bitmap 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore block size = 96636764160 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore fsck on mount = True 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluefs = 1/20 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluestore = 1/20 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-08T08:28:16.207 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: debug rocksdb = 4/10 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd backfillfull_ratio = 0.85 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd full ratio = 0.9 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd nearfull ratio = 0.8 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: osd failsafe full ratio = 0.95 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = bluestore 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-08T08:28:16.208 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2024-03-08T08:28:16.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-08T08:28:16.210 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = "" 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds balancer = 20 2024-03-08T08:28:16.211 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-03-08T08:28:16.212 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-03-08T08:28:16.213 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-08T08:28:16.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-08T08:28:16.308 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-08T08:28:16.309 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:16.309 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-08T08:28:16.322 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:16.323 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-08T08:28:16.377 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:28:16.377 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:16.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-08T08:28:16.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-08T08:28:16.455 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:16.455 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:16.510 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:28:16.510 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:16.510 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-08T08:28:16.511 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-08T08:28:16.551 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:16.551 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:16.606 INFO:tasks.ceph:Setting up mon nodes... 2024-03-08T08:28:16.606 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-08T08:28:16.606 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-08T08:28:16.660 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-08T08:28:16.663 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-08T08:28:16.720 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-08T08:28:16.723 INFO:tasks.ceph:Setting up mds nodes... 2024-03-08T08:28:16.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-08T08:28:16.777 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-08T08:28:16.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-08T08:28:16.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-08T08:28:16.899 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-08T08:28:16.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-08T08:28:16.928 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-08T08:28:16.983 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-08T08:28:16.985 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-08T08:28:17.012 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-08T08:28:17.106 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-08T08:28:17.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-08T08:28:17.135 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-08T08:28:17.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-08T08:28:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-08T08:28:17.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-08T08:28:17.274 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-08T08:28:17.289 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-08T08:28:17.331 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-08T08:28:17.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-08T08:28:17.428 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-08T08:28:17.443 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-08T08:28:17.443 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-08T08:28:17.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-08T08:28:17.470 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:17.470 INFO:tasks.ceph:role: osd.0 2024-03-08T08:28:17.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:28:17.470 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:17.546 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:18.051 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-08T08:28:18.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-08T08:28:18.054 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-08T08:28:18.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-08T08:28:18.219 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-08T08:28:18.286 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:18.286 INFO:tasks.ceph:role: osd.1 2024-03-08T08:28:18.286 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:28:18.287 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-08T08:28:18.361 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:18.362 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:18.809 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-08T08:28:18.811 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-08T08:28:18.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-08T08:28:18.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-08T08:28:18.947 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-08T08:28:19.016 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:19.016 INFO:tasks.ceph:role: osd.2 2024-03-08T08:28:19.016 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:28:19.016 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-08T08:28:19.092 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:19.092 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:19.092 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:19.093 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:19.580 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-08T08:28:19.581 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-08T08:28:19.582 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-08T08:28:19.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-08T08:28:19.747 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-08T08:28:19.813 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:19.813 INFO:tasks.ceph:role: osd.3 2024-03-08T08:28:19.813 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:28:19.814 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-08T08:28:19.887 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:19.888 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:20.392 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-08T08:28:20.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-08T08:28:20.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-08T08:28:20.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-08T08:28:20.555 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.656+0000 7f8a4a41e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.656+0000 7f8a4a41e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.656+0000 7f8a4a41e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.656+0000 7f8a4a41e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.656+0000 7f8a4a41e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-08T08:28:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:20.657+0000 7f8a4a41e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-08T08:28:23.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-08T08:28:23.539 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:23.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.628+0000 7f294548b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-08T08:28:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.628+0000 7f294548b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-08T08:28:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.628+0000 7f294548b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-08T08:28:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.628+0000 7f294548b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-08T08:28:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.628+0000 7f294548b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-08T08:28:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:23.629+0000 7f294548b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-08T08:28:26.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-08T08:28:26.507 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.596+0000 7f3ac8d14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-08T08:28:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.596+0000 7f3ac8d14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-08T08:28:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.596+0000 7f3ac8d14640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-08T08:28:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.596+0000 7f3ac8d14640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-08T08:28:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.596+0000 7f3ac8d14640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-08T08:28:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:26.597+0000 7f3ac8d14640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-08T08:28:29.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-08T08:28:29.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.562+0000 7f1f90615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-08T08:28:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.562+0000 7f1f90615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-08T08:28:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.562+0000 7f1f90615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-08T08:28:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.562+0000 7f1f90615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-08T08:28:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.562+0000 7f1f90615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-08T08:28:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:29.563+0000 7f1f90615640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-08T08:28:32.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-08T08:28:32.460 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-08T08:28:32.493 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:32.493 INFO:tasks.ceph:role: osd.4 2024-03-08T08:28:32.493 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:28:32.493 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-08T08:28:32.563 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:32.564 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:32.597 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-08T08:28:32.599 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-08T08:28:32.599 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-08T08:28:32.678 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-08T08:28:32.749 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-08T08:28:32.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:32.816 INFO:tasks.ceph:role: osd.5 2024-03-08T08:28:32.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:28:32.816 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:32.883 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:32.915 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-08T08:28:32.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-08T08:28:32.917 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-08T08:28:32.987 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-08T08:28:33.058 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-08T08:28:33.124 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:33.125 INFO:tasks.ceph:role: osd.6 2024-03-08T08:28:33.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:28:33.125 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-08T08:28:33.190 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:33.190 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:33.190 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:33.190 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:33.190 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:33.191 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:33.191 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:33.191 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:33.191 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:33.191 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:33.223 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-08T08:28:33.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-08T08:28:33.225 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-08T08:28:33.294 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-08T08:28:33.365 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-08T08:28:33.432 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-08T08:28:33.432 INFO:tasks.ceph:role: osd.7 2024-03-08T08:28:33.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:28:33.433 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-08T08:28:33.499 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-08T08:28:33.531 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-08T08:28:33.532 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-08T08:28:33.533 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-08T08:28:33.607 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-08T08:28:33.678 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.769+0000 7fa4de41e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.770+0000 7fa4de41e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.770+0000 7fa4de41e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.770+0000 7fa4de41e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.770+0000 7fa4de41e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-08T08:28:33.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:33.770+0000 7fa4de41e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-08T08:28:36.635 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-08T08:28:36.666 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-08T08:28:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:36.757+0000 7fc7aa7bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-08T08:28:39.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-08T08:28:39.635 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:39.726 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-08T08:28:39.726 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-08T08:28:39.726 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-08T08:28:39.727 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-08T08:28:39.727 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-08T08:28:39.727 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:39.724+0000 7fa351824640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-08T08:28:42.573 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-08T08:28:42.604 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:42.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.695+0000 7fe802e24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-08T08:28:42.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.695+0000 7fe802e24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-08T08:28:42.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.695+0000 7fe802e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-08T08:28:42.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.695+0000 7fe802e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-08T08:28:42.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.695+0000 7fe802e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-08T08:28:42.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:42.696+0000 7fe802e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-08T08:28:44.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:44.214 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:28:45.540 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-08T08:28:45.571 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-08T08:28:45.571 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.571 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-08T08:28:45.603 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.603 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-08T08:28:45.674 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-08T08:28:45.743 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-08T08:28:45.813 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-08T08:28:45.881 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-08T08:28:45.948 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:45.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-08T08:28:46.015 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-08T08:28:46.142 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.142 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-08T08:28:46.211 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.212 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-08T08:28:46.281 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.281 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-08T08:28:46.349 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.349 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-08T08:28:46.416 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.416 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-08T08:28:46.484 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.484 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-08T08:28:46.550 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:46.550 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-08T08:28:46.568 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:46.568 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-08T08:28:46.625 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.625 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-08T08:28:46.639 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:46.639 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-08T08:28:46.694 INFO:tasks.ceph:Adding keys to all mons... 2024-03-08T08:28:46.694 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-08T08:28:46.697 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-08T08:28:46.737 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.y] 2024-03-08T08:28:46.737 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAgzOplSUhfJxAAywlcJde+D6Ms0WSzgWojlw== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAgzOplDpdVLhAAo1qwVTByZkCW99JvxNG5fw== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAgzOplAymcNRAA0wcrVZraXrP1J4XjBgfVqQ== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAkzOplpscuJxAAvVJ1o5YDhHfY4zE6BlFAyg== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAnzOpl6LCTJRAApue+ew4o4RVwgj56sgHhuw== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAqzOpl1uikIxAAmk+OecUZ40V+w7kBueFp8g== 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2024-03-08T08:28:46.738 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAtzOplwfSjIRAAbrGyWTJRwqcwqktutrXUnA== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAgzOplSrTyKhAAHqlsN/aGIbJH4WlA/120cA== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAgzOplDwOYOhAAZZCYzARU9SC14+aUfg4upA== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplqRpYBhAAfgXmaCR/NePZNhmJNzSbxQ== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAxzOplnpH8LRAAFcm5yk7qNJEQNUoq9VWymQ== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA0zOplRRY5LRAAmOK8521OeEEjxKdzdSylxg== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.6] 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA3zOplpQtHKxAALqvEWOSsbexSBWKTehdREQ== 2024-03-08T08:28:46.739 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.7] 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA6zOplwMqQKRAAJNPW34xRBzrJP2qTwCQITQ== 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplr8ebChAA8n4wUJZpCY0XA2qC5ELxpw== 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout:[client.2] 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplJi9bEBAA8Xr6QHVKC5Spv8RLRKCrxw== 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplCj7FExAADQsys35++1vKbNpqu8Sigw== 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout:[client.3] 2024-03-08T08:28:46.740 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplcTeHGRAAvXhoVFl4INMTIU09S114jw== 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.y] 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAgzOplSUhfJxAAywlcJde+D6Ms0WSzgWojlw== 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAgzOplDpdVLhAAo1qwVTByZkCW99JvxNG5fw== 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAgzOplAymcNRAA0wcrVZraXrP1J4XjBgfVqQ== 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAkzOplpscuJxAAvVJ1o5YDhHfY4zE6BlFAyg== 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2024-03-08T08:28:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAnzOpl6LCTJRAApue+ew4o4RVwgj56sgHhuw== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAqzOpl1uikIxAAmk+OecUZ40V+w7kBueFp8g== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.3] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAtzOplwfSjIRAAbrGyWTJRwqcwqktutrXUnA== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAgzOplSrTyKhAAHqlsN/aGIbJH4WlA/120cA== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAgzOplDwOYOhAAZZCYzARU9SC14+aUfg4upA== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAhzOplqRpYBhAAfgXmaCR/NePZNhmJNzSbxQ== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.4] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAxzOplnpH8LRAAFcm5yk7qNJEQNUoq9VWymQ== 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.5] 2024-03-08T08:28:46.761 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA0zOplRRY5LRAAmOK8521OeEEjxKdzdSylxg== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.6] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA3zOplpQtHKxAALqvEWOSsbexSBWKTehdREQ== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.7] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA6zOplwMqQKRAAJNPW34xRBzrJP2qTwCQITQ== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAhzOplr8ebChAA8n4wUJZpCY0XA2qC5ELxpw== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[client.2] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAhzOplJi9bEBAA8Xr6QHVKC5Spv8RLRKCrxw== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAhzOplCj7FExAADQsys35++1vKbNpqu8Sigw== 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout:[client.3] 2024-03-08T08:28:46.762 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAhzOplcTeHGRAAvXhoVFl4INMTIU09S114jw== 2024-03-08T08:28:46.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-08T08:28:46.784 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-08T08:28:46.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:46.857 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:46.912 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:46.914 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:46.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:46.971 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.024 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.078 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.129 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.131 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-08T08:28:47.185 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-08T08:28:47.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:47.238 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:47.293 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:47.296 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-08T08:28:47.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.352 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.404 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.460 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.511 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.513 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-08T08:28:47.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.567 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.678 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.681 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-08T08:28:47.792 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-08T08:28:47.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-08T08:28:47.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-08T08:28:47.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-08T08:28:47.939 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-08T08:28:47.967 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-08T08:28:48.063 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-08T08:28:48.089 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-08T08:28:48.156 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-08T08:28:48.252 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-08T08:28:48.279 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:48.282 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-08T08:28:48.334 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-08T08:28:48.334 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-08T08:28:48.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-08T08:28:48.337 INFO:tasks.ceph.mon.a:Started 2024-03-08T08:28:48.337 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-08T08:28:48.337 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-08T08:28:48.376 INFO:tasks.ceph.mon.b:Started 2024-03-08T08:28:48.376 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-08T08:28:48.376 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-08T08:28:48.378 INFO:tasks.ceph.mon.c:Started 2024-03-08T08:28:48.378 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-08T08:28:48.378 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-08T08:28:48.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-08T08:28:48.380 INFO:tasks.ceph.mgr.y:Started 2024-03-08T08:28:48.380 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-08T08:28:48.380 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-08T08:28:48.382 INFO:tasks.ceph.mgr.x:Started 2024-03-08T08:28:48.383 INFO:tasks.ceph:Setting crush tunables to default 2024-03-08T08:28:48.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2024-03-08T08:28:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.482+0000 7f843def4640 1 Processor -- start 2024-03-08T08:28:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.482+0000 7f843def4640 1 -- start start 2024-03-08T08:28:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f843def4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 0x7f843815e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f843def4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f843def4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84380c7250 con 0x7f843815ae70 2024-03-08T08:28:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f843def4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843815c490 con 0x7f84380c5c80 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f843def4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843815eea0 con 0x7f843815e580 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47600/0 (socket says 172.21.15.12:47600) 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 learned_addr learned my addr 172.21.15.12:0/3506012493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f84377fe640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 0x7f843815e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 msgr2=0x7f84380c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 msgr2=0x7f84380c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.483+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.484+0000 7f84377fe640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 msgr2=0x7f843815e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.484+0000 7f84377fe640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 msgr2=0x7f843815e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.484+0000 7f84377fe640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 0x7f843815e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-08T08:28:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.484+0000 7f84377fe640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 0x7f843815e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T08:28:48.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.672+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297273453 0 0) 0x7f84380c7250 con 0x7f843815ae70 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.672+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8420003520 con 0x7f843815ae70 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802825251 0 0) 0x7f8420003520 con 0x7f843815ae70 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 msgr2=0x7f843815e960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.156:3300/0 conn(0x7f843815e580 0x7f843815e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 msgr2=0x7f84380c60e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 --2- 172.21.15.12:0/3506012493 >> v2:172.21.15.12:3300/0 conn(0x7f84380c5c80 0x7f84380c60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380c69b0 con 0x7f843815ae70 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f84300019f0 con 0x7f843815ae70 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 >> v1:172.21.15.156:6789/0 conn(0x7f843815ae70 legacy=0x7f843815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84200039b0 0x7f8420003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 0x7f8420004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84200051c0 0x7f842000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843815c490 con 0x7f8420004800 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843815eea0 con 0x7f84200051c0 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842000c170 con 0x7f84200039b0 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 0x7f8420004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f84377fe640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84200039b0 0x7f8420003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f8437fff640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84200051c0 0x7f842000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84200039b0 msgr2=0x7f8420003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.673+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84200039b0 0x7f8420003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.674+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84200051c0 msgr2=0x7f842000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.674+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84200051c0 0x7f842000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.674+0000 7f8437fff640 1 -- 172.21.15.12:0/3506012493 reap_dead start 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.674+0000 7f8436ffd640 1 -- 172.21.15.12:0/3506012493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420003520 con 0x7f8420004800 2024-03-08T08:28:48.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.675+0000 7f8436ffd640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 0x7f8420004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f84200055a0 tx=0x7f8420005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841edeaffce2c1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.675+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c00f040 con 0x7f8420004800 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.675+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c037070 con 0x7f8420004800 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.675+0000 7f84367fc640 1 -- 172.21.15.12:0/3506012493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0328a0 con 0x7f8420004800 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 -- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 msgr2=0x7f8420004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 0x7f8420004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f84200055a0 tx=0x7f8420005830 comp rx=0 tx=0).stop 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 -- 172.21.15.12:0/3506012493 shutdown_connections 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 --2- 172.21.15.12:0/3506012493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8420004800 0x7f8420004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 -- 172.21.15.12:0/3506012493 >> 172.21.15.12:0/3506012493 conn(0x7f8438099a40 msgr2=0x7f8438099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 -- 172.21.15.12:0/3506012493 shutdown_connections 2024-03-08T08:28:48.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.676+0000 7f843def4640 1 -- 172.21.15.12:0/3506012493 wait complete. 2024-03-08T08:28:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.678+0000 7f843def4640 1 Processor -- start 2024-03-08T08:28:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.678+0000 7f843def4640 1 -- start start 2024-03-08T08:28:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84380cae10 0x7f843815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84380b0060 con 0x7f84380cb290 2024-03-08T08:28:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84380c6f70 con 0x7f8420004800 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f843def4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84380b01e0 con 0x7f84380cae10 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f8437fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f84377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f84377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:47584/0 (socket says 172.21.15.12:47584) 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f8437fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47646/0 (socket says 172.21.15.12:47646) 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f84377fe640 1 -- 172.21.15.12:0/651528342 learned_addr learned my addr 172.21.15.12:0/651528342 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.679+0000 7f8436ffd640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84380cae10 0x7f843815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 -- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84380cae10 msgr2=0x7f843815d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84380cae10 0x7f843815d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 -- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 msgr2=0x7f8438149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 -- 172.21.15.12:0/651528342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438149950 con 0x7f84380cb290 2024-03-08T08:28:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f84377fe640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f8437fff640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8438067920 tx=0x7f843000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430003b70 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430003d70 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84381f8040 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300052b0 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84381f74c0 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.680+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8430005450 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.681+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8430011780 con 0x7f84380cb290 2024-03-08T08:28:48.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.681+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84380749c0 con 0x7f84380cb290 2024-03-08T08:28:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.684+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f843001dce0 con 0x7f84380cb290 2024-03-08T08:28:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.752+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8438073170 con 0x7f84380cb290 2024-03-08T08:28:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.753+0000 7f841ffff640 1 -- 172.21.15.12:0/651528342 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f843001d470 con 0x7f84380cb290 2024-03-08T08:28:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-03-08T08:28:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 msgr2=0x7f843815db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8438067920 tx=0x7f843000f730 comp rx=0 tx=0).stop 2024-03-08T08:28:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 shutdown_connections 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84380cb290 0x7f843815db70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84380cae10 0x7f843815d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 --2- 172.21.15.12:0/651528342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8420004800 0x7f8438149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.754+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 >> 172.21.15.12:0/651528342 conn(0x7f8438099a40 msgr2=0x7f843806abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.755+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 shutdown_connections 2024-03-08T08:28:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:48.755+0000 7f843def4640 1 -- 172.21.15.12:0/651528342 wait complete. 2024-03-08T08:28:48.770 INFO:tasks.ceph:check_enable_crimson: False 2024-03-08T08:28:48.770 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-08T08:28:48.771 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:48.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-08T08:28:48.797 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:48.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-08T08:28:48.864 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:48.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-08T08:28:48.931 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:28:48.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-08T08:28:48.998 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:48.998 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-08T08:28:49.025 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:49.025 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-08T08:28:49.091 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:49.092 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-08T08:28:49.159 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:28:49.159 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-08T08:28:49.160 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.159+0000 7fc5dc847040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T08:28:49.215 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.214+0000 7fc5dc847040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T08:28:49.226 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new b18ed71d-fb54-456e-b684-4eed2d74d441 0 2024-03-08T08:28:49.247 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:49.245+0000 7fb305639040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T08:28:49.304 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:49.303+0000 7fb305639040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T08:28:49.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.345+0000 7ff88631e640 1 Processor -- start 2024-03-08T08:28:49.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.345+0000 7ff88631e640 1 -- start start 2024-03-08T08:28:49.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff88631e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff88015e6d0 0x7ff880070090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff88631e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8800cbb90 0x7ff8800cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff88631e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88006f840 con 0x7ff88015f2e0 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff88631e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800cdd10 con 0x7ff8800cbb90 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff88631e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800705d0 con 0x7ff88015e6d0 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff88015e6d0 0x7ff880070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff88015e6d0 0x7ff880070090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:44644/0 (socket says 172.21.15.156:44644) 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 -- 172.21.15.156:0/4243751282 learned_addr learned my addr 172.21.15.156:0/4243751282 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff8848bb640 1 --2- 172.21.15.156:0/4243751282 >> v2:172.21.15.12:3300/0 conn(0x7ff8800cbb90 0x7ff8800cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 -- 172.21.15.156:0/4243751282 >> v2:172.21.15.12:3300/0 conn(0x7ff8800cbb90 msgr2=0x7ff8800cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 --2- 172.21.15.156:0/4243751282 >> v2:172.21.15.12:3300/0 conn(0x7ff8800cbb90 0x7ff8800cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 -- 172.21.15.156:0/4243751282 >> v1:172.21.15.156:6789/0 conn(0x7ff88015f2e0 legacy=0x7ff88015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.346+0000 7ff87ffff640 1 -- 172.21.15.156:0/4243751282 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88006fb20 con 0x7ff88015e6d0 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.347+0000 7ff87ffff640 1 --2- 172.21.15.156:0/4243751282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff880070090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8800733c0 tx=0x7ff87402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c11049a9246a1c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.347+0000 7ff87effd640 1 -- 172.21.15.156:0/4243751282 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874034070 con 0x7ff88015e6d0 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.347+0000 7ff87effd640 1 -- 172.21.15.156:0/4243751282 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff874030860 con 0x7ff88015e6d0 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.347+0000 7ff87effd640 1 -- 172.21.15.156:0/4243751282 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874030c30 con 0x7ff88015e6d0 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 -- 172.21.15.156:0/4243751282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 msgr2=0x7ff880070090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 --2- 172.21.15.156:0/4243751282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff880070090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8800733c0 tx=0x7ff87402b9a0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 -- 172.21.15.156:0/4243751282 shutdown_connections 2024-03-08T08:28:49.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 --2- 172.21.15.156:0/4243751282 >> v2:172.21.15.12:3300/0 conn(0x7ff8800cbb90 0x7ff8800cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 --2- 172.21.15.156:0/4243751282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff880070090 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 -- 172.21.15.156:0/4243751282 >> 172.21.15.156:0/4243751282 conn(0x7ff880099a40 msgr2=0x7ff8800b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 -- 172.21.15.156:0/4243751282 shutdown_connections 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.348+0000 7ff88631e640 1 -- 172.21.15.156:0/4243751282 wait complete. 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.349+0000 7ff88631e640 1 Processor -- start 2024-03-08T08:28:49.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.349+0000 7ff88631e640 1 -- start start 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8800cbb90 0x7ff8800c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 0x7ff8800c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800cb390 con 0x7ff8800cbb90 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800cb700 con 0x7ff88015e6d0 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff88631e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88006f560 con 0x7ff88015f2e0 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff8848bb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 0x7ff8800c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:44654/0 (socket says 172.21.15.156:44654) 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 -- 172.21.15.156:0/3664364604 learned_addr learned my addr 172.21.15.156:0/3664364604 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87ffff640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8800cbb90 0x7ff8800c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 -- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 msgr2=0x7ff8800c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 0x7ff8800c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 -- 172.21.15.156:0/3664364604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8800cbb90 msgr2=0x7ff8800c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.350+0000 7ff87f7fe640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8800cbb90 0x7ff8800c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87f7fe640 1 -- 172.21.15.156:0/3664364604 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8801eff90 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff8848bb640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 0x7ff8800c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87f7fe640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff87000c440 tx=0x7ff870001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8700052d0 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff870005470 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8801f0270 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff870007260 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8801f0b10 con 0x7ff88015e6d0 2024-03-08T08:28:49.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff870007400 con 0x7ff88015e6d0 2024-03-08T08:28:49.357 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.351+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff870003b30 con 0x7ff88015e6d0 2024-03-08T08:28:49.357 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.352+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8801f0660 con 0x7ff88015e6d0 2024-03-08T08:28:49.357 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.354+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff870003d80 con 0x7ff88015e6d0 2024-03-08T08:28:49.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.425+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b18ed71d-fb54-456e-b684-4eed2d74d441", "id": 0} v 0) v1 -- 0x7ff8801f0440 con 0x7ff88015e6d0 2024-03-08T08:28:49.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff87cff9640 1 -- 172.21.15.156:0/3664364604 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b18ed71d-fb54-456e-b684-4eed2d74d441", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff8700035f0 con 0x7ff88015e6d0 2024-03-08T08:28:49.430 INFO:teuthology.orchestra.run.smithi156.stdout:0 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 msgr2=0x7ff8800c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff87000c440 tx=0x7ff870001e70 comp rx=0 tx=0).stop 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 shutdown_connections 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff88015f2e0 0x7ff8800c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff88015e6d0 0x7ff8800c60c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 --2- 172.21.15.156:0/3664364604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8800cbb90 0x7ff8800c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.428+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 >> 172.21.15.156:0/3664364604 conn(0x7ff880099a40 msgr2=0x7ff88015c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.429+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 shutdown_connections 2024-03-08T08:28:49.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.429+0000 7ff88631e640 1 -- 172.21.15.156:0/3664364604 wait complete. 2024-03-08T08:28:49.441 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 1a7daaf5-7d2b-4593-8fb6-7f73346f63a4 1 2024-03-08T08:28:49.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.540+0000 7f29b1883640 1 Processor -- start 2024-03-08T08:28:49.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.540+0000 7f29b1883640 1 -- start start 2024-03-08T08:28:49.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29b1883640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29b1883640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29b1883640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac15be80 con 0x7f29ac06abd0 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29b1883640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0cdd20 con 0x7f29ac0cd380 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29b1883640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac06f3b0 con 0x7f29ac15baa0 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29aaffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29aa7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29aaffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:39588/0 (socket says 172.21.15.156:39588) 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29aaffd640 1 -- 172.21.15.156:0/3933400927 learned_addr learned my addr 172.21.15.156:0/3933400927 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.541+0000 7f29aa7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:44666/0 (socket says 172.21.15.156:44666) 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aaffd640 1 -- 172.21.15.156:0/3933400927 >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 msgr2=0x7f29ac0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aaffd640 1 --2- 172.21.15.156:0/3933400927 >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aaffd640 1 -- 172.21.15.156:0/3933400927 >> v1:172.21.15.156:6789/0 conn(0x7f29ac06abd0 legacy=0x7f29ac06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aaffd640 1 -- 172.21.15.156:0/3933400927 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac06f690 con 0x7f29ac15baa0 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aa7fc640 1 --2- 172.21.15.156:0/3933400927 >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.542+0000 7f29aaffd640 1 --2- 172.21.15.156:0/3933400927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009870 tx=0x7f29a002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69b590207772d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29a9ffb640 1 -- 172.21.15.156:0/3933400927 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0034070 con 0x7f29ac15baa0 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29a9ffb640 1 -- 172.21.15.156:0/3933400927 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0030840 con 0x7f29ac15baa0 2024-03-08T08:28:49.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29a9ffb640 1 -- 172.21.15.156:0/3933400927 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0030c10 con 0x7f29ac15baa0 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 -- 172.21.15.156:0/3933400927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 msgr2=0x7f29ac06bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 --2- 172.21.15.156:0/3933400927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009870 tx=0x7f29a002b980 comp rx=0 tx=0).stop 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 -- 172.21.15.156:0/3933400927 shutdown_connections 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 --2- 172.21.15.156:0/3933400927 >> v2:172.21.15.156:3300/0 conn(0x7f29ac0cd380 0x7f29ac0cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 --2- 172.21.15.156:0/3933400927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac15baa0 0x7f29ac06bf10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 -- 172.21.15.156:0/3933400927 >> 172.21.15.156:0/3933400927 conn(0x7f29ac06db00 msgr2=0x7f29ac06df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 -- 172.21.15.156:0/3933400927 shutdown_connections 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.543+0000 7f29b1883640 1 -- 172.21.15.156:0/3933400927 wait complete. 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.544+0000 7f29b1883640 1 Processor -- start 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 -- start start 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29ac06abd0 0x7f29ac141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29ac15baa0 0x7f29ac1e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac06ff30 con 0x7f29ac06abd0 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac06fc50 con 0x7f29ac15baa0 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29b1883640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac06f040 con 0x7f29ac0cd380 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29aa7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29ab7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29ac15baa0 0x7f29ac1e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29aa7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:39590/0 (socket says 172.21.15.156:39590) 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.545+0000 7f29aa7fc640 1 -- 172.21.15.156:0/3748767904 learned_addr learned my addr 172.21.15.156:0/3748767904 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aaffd640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29ac06abd0 0x7f29ac141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 -- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29ac15baa0 msgr2=0x7f29ac1e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29ac15baa0 0x7f29ac1e4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 -- 172.21.15.156:0/3748767904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29ac06abd0 msgr2=0x7f29ac141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29ac06abd0 0x7f29ac141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 -- 172.21.15.156:0/3748767904 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac142160 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29aa7fc640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f299c00ec10 tx=0x7f299c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00cc30 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c00cdd0 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac142440 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c005bb0 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac142a00 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f299c0063c0 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.546+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f299c013cc0 con 0x7f29ac0cd380 2024-03-08T08:28:49.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.547+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac0749c0 con 0x7f29ac0cd380 2024-03-08T08:28:49.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.549+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f299c017020 con 0x7f29ac0cd380 2024-03-08T08:28:49.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.621+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1a7daaf5-7d2b-4593-8fb6-7f73346f63a4", "id": 1} v 0) v1 -- 0x7f29ac15cad0 con 0x7f29ac0cd380 2024-03-08T08:28:49.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.624+0000 7f2993fff640 1 -- 172.21.15.156:0/3748767904 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a7daaf5-7d2b-4593-8fb6-7f73346f63a4", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f299c013470 con 0x7f29ac0cd380 2024-03-08T08:28:49.626 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 msgr2=0x7f29ac143200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f299c00ec10 tx=0x7f299c00c520 comp rx=0 tx=0).stop 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 shutdown_connections 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29ac15baa0 0x7f29ac1e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29ac0cd380 0x7f29ac143200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 --2- 172.21.15.156:0/3748767904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29ac06abd0 0x7f29ac141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.625+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 >> 172.21.15.156:0/3748767904 conn(0x7f29ac06db00 msgr2=0x7f29ac15c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.626+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 shutdown_connections 2024-03-08T08:28:49.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.626+0000 7f29b1883640 1 -- 172.21.15.156:0/3748767904 wait complete. 2024-03-08T08:28:49.638 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new f11ad7d7-d93b-4dbf-9c65-91a9433a5095 2 2024-03-08T08:28:49.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 Processor -- start 2024-03-08T08:28:49.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 -- start start 2024-03-08T08:28:49.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815dec0 0x7fda58161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 0x7fda580d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580b6be0 con 0x7fda580d0790 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d2140 con 0x7fda58161ed0 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda5d4bb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5815e2a0 con 0x7fda5815dec0 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.738+0000 7fda56ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815dec0 0x7fda58161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815dec0 0x7fda58161990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50080/0 (socket says 172.21.15.156:50080) 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 -- 172.21.15.156:0/3681278571 learned_addr learned my addr 172.21.15.156:0/3681278571 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda567fc640 1 --2- 172.21.15.156:0/3681278571 >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 0x7fda580d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 -- 172.21.15.156:0/3681278571 >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 msgr2=0x7fda580d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 --2- 172.21.15.156:0/3681278571 >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 0x7fda580d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 -- 172.21.15.156:0/3681278571 >> v1:172.21.15.156:6789/0 conn(0x7fda580d0790 legacy=0x7fda580b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda56ffd640 1 -- 172.21.15.156:0/3681278571 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda58162350 con 0x7fda5815dec0 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.739+0000 7fda567fc640 1 --2- 172.21.15.156:0/3681278571 >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 0x7fda580d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda56ffd640 1 --2- 172.21.15.156:0/3681278571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58161990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda580b6640 tx=0x7fda4802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2861865d93f4214 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda55ffb640 1 -- 172.21.15.156:0/3681278571 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48034070 con 0x7fda5815dec0 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda55ffb640 1 -- 172.21.15.156:0/3681278571 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48030860 con 0x7fda5815dec0 2024-03-08T08:28:49.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda55ffb640 1 -- 172.21.15.156:0/3681278571 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48030c30 con 0x7fda5815dec0 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3681278571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 msgr2=0x7fda58161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3681278571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58161990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda580b6640 tx=0x7fda4802b9a0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3681278571 shutdown_connections 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3681278571 >> v2:172.21.15.12:3300/0 conn(0x7fda58161ed0 0x7fda580d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3681278571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58161990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3681278571 >> 172.21.15.156:0/3681278571 conn(0x7fda5806c240 msgr2=0x7fda5806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3681278571 shutdown_connections 2024-03-08T08:28:49.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.740+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3681278571 wait complete. 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 Processor -- start 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 -- start start 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 0x7fda581f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda58161ed0 0x7fda580cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d0bd0 con 0x7fda58161ed0 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d1dd0 con 0x7fda5815dec0 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda5d4bb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b94f0 con 0x7fda580d0790 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda567fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.742+0000 7fda56ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 0x7fda581f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda567fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50084/0 (socket says 172.21.15.156:50084) 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda567fc640 1 -- 172.21.15.156:0/3522820972 learned_addr learned my addr 172.21.15.156:0/3522820972 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda577fe640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda58161ed0 0x7fda580cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 -- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 msgr2=0x7fda58159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 -- 172.21.15.156:0/3522820972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda58161ed0 msgr2=0x7fda580cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda58161ed0 0x7fda580cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 -- 172.21.15.156:0/3522820972 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580c97c0 con 0x7fda580d0790 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda567fc640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda56ffd640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 0x7fda581f49f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda48033000 tx=0x7fda48004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48048070 con 0x7fda580d0790 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48030660 con 0x7fda580d0790 2024-03-08T08:28:49.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda580c9aa0 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48034040 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda580ca060 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fda4804b070 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.743+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fda48043830 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.744+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda580749c0 con 0x7fda580d0790 2024-03-08T08:28:49.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.747+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fda48052070 con 0x7fda580d0790 2024-03-08T08:28:49.774 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.773+0000 7fc5dc847040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T08:28:49.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.819+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f11ad7d7-d93b-4dbf-9c65-91a9433a5095", "id": 2} v 0) v1 -- 0x7fda58161450 con 0x7fda580d0790 2024-03-08T08:28:49.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.825+0000 7fda3f7fe640 1 -- 172.21.15.156:0/3522820972 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f11ad7d7-d93b-4dbf-9c65-91a9433a5095", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fda48030830 con 0x7fda580d0790 2024-03-08T08:28:49.827 INFO:teuthology.orchestra.run.smithi156.stdout:2 2024-03-08T08:28:49.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 msgr2=0x7fda581f49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 0x7fda581f49f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda48033000 tx=0x7fda48004730 comp rx=0 tx=0).stop 2024-03-08T08:28:49.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 shutdown_connections 2024-03-08T08:28:49.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda58161ed0 0x7fda580cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815dec0 0x7fda58159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 --2- 172.21.15.156:0/3522820972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda580d0790 0x7fda581f49f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 >> 172.21.15.156:0/3522820972 conn(0x7fda5806c240 msgr2=0x7fda580b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 shutdown_connections 2024-03-08T08:28:49.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.826+0000 7fda5d4bb640 1 -- 172.21.15.156:0/3522820972 wait complete. 2024-03-08T08:28:49.831 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.829+0000 7fc5dc847040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T08:28:49.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new f3e67c25-59c6-4c91-aa05-7e2310962ced 3 2024-03-08T08:28:49.877 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.876+0000 7fc5dc847040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T08:28:49.881 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:49.879+0000 7fb305639040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T08:28:49.932 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.931+0000 7fc5dc847040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T08:28:49.938 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:49.936+0000 7fb305639040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T08:28:49.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.938+0000 7f4ea88ae640 1 Processor -- start 2024-03-08T08:28:49.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.938+0000 7f4ea88ae640 1 -- start start 2024-03-08T08:28:49.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 0x7f4ea40cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea415c090 con 0x7f4ea40c5a30 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea40cd7a0 con 0x7f4ea409a9c0 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea88ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4099c00 con 0x7f4ea415bcb0 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea1d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea1d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50094/0 (socket says 172.21.15.156:50094) 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:41988/0 (socket says 172.21.15.156:41988) 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea1d9b640 1 -- 172.21.15.156:0/2447667092 learned_addr learned my addr 172.21.15.156:0/2447667092 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea2d9d640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 0x7f4ea40cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 -- 172.21.15.156:0/2447667092 >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 msgr2=0x7f4ea40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:49.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 -- 172.21.15.156:0/2447667092 >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 msgr2=0x7f4ea40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:49.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 0x7f4ea40cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea259c640 1 -- 172.21.15.156:0/2447667092 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4099ee0 con 0x7f4ea415bcb0 2024-03-08T08:28:49.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea1d9b640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:49.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.939+0000 7f4ea2d9d640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 0x7f4ea40cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.940+0000 7f4ea259c640 1 --2- 172.21.15.156:0/2447667092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4ea406bbb0 tx=0x7f4e9802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fde9cca80964a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.940+0000 7f4ea159a640 1 -- 172.21.15.156:0/2447667092 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98038070 con 0x7f4ea415bcb0 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.940+0000 7f4ea159a640 1 -- 172.21.15.156:0/2447667092 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98031700 con 0x7f4ea415bcb0 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.940+0000 7f4ea159a640 1 -- 172.21.15.156:0/2447667092 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98031ad0 con 0x7f4ea415bcb0 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 -- 172.21.15.156:0/2447667092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 msgr2=0x7f4ea40ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/2447667092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4ea406bbb0 tx=0x7f4e9802b7a0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 -- 172.21.15.156:0/2447667092 shutdown_connections 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.12:3300/0 conn(0x7f4ea40c5a30 0x7f4ea40cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/2447667092 >> v2:172.21.15.156:3300/0 conn(0x7f4ea409a9c0 0x7f4ea40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/2447667092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea415bcb0 0x7f4ea40ccf70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 -- 172.21.15.156:0/2447667092 >> 172.21.15.156:0/2447667092 conn(0x7f4ea40b3490 msgr2=0x7f4ea40b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 -- 172.21.15.156:0/2447667092 shutdown_connections 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.941+0000 7f4ea88ae640 1 -- 172.21.15.156:0/2447667092 wait complete. 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.942+0000 7f4ea88ae640 1 Processor -- start 2024-03-08T08:28:49.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.942+0000 7f4ea88ae640 1 -- start start 2024-03-08T08:28:49.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea40c5a30 0x7f4ea41575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ea415bcb0 0x7f4ea40cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:49.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40cd4b0 con 0x7f4ea40c5a30 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea409a4a0 con 0x7f4ea415bcb0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea88ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea409ae20 con 0x7f4ea409a9c0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea2d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ea415bcb0 0x7f4ea40cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:41990/0 (socket says 172.21.15.156:41990) 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 -- 172.21.15.156:0/3207312236 learned_addr learned my addr 172.21.15.156:0/3207312236 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea1d9b640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea40c5a30 0x7f4ea41575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 -- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ea415bcb0 msgr2=0x7f4ea40cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ea415bcb0 0x7f4ea40cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 -- 172.21.15.156:0/3207312236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea40c5a30 msgr2=0x7f4ea41575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea40c5a30 0x7f4ea41575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.943+0000 7f4ea259c640 1 -- 172.21.15.156:0/3207312236 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4158230 con 0x7f4ea409a9c0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4ea259c640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e98000c00 tx=0x7f4e980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:49.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98042070 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9803d870 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea4158510 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98038040 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea4158ad0 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4e98045070 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.944+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f4e98047de0 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.945+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea40749c0 con 0x7f4ea409a9c0 2024-03-08T08:28:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:49.947+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4e98030070 con 0x7f4ea409a9c0 2024-03-08T08:28:49.977 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:49.976+0000 7fc5dc847040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T08:28:49.986 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:49.984+0000 7fb305639040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T08:28:50.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.020+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f3e67c25-59c6-4c91-aa05-7e2310962ced", "id": 3} v 0) v1 -- 0x7f4ea40c5ef0 con 0x7f4ea409a9c0 2024-03-08T08:28:50.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.025+0000 7f4e92ffd640 1 -- 172.21.15.156:0/3207312236 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3e67c25-59c6-4c91-aa05-7e2310962ced", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4e98047450 con 0x7f4ea409a9c0 2024-03-08T08:28:50.028 INFO:teuthology.orchestra.run.smithi156.stdout:3 2024-03-08T08:28:50.028 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 msgr2=0x7f4ea40b2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e98000c00 tx=0x7f4e980046c0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 shutdown_connections 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ea415bcb0 0x7f4ea40cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea40c5a30 0x7f4ea41575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 --2- 172.21.15.156:0/3207312236 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea409a9c0 0x7f4ea40b2930 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 >> 172.21.15.156:0/3207312236 conn(0x7f4ea40b3490 msgr2=0x7f4ea40cf900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 shutdown_connections 2024-03-08T08:28:50.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.026+0000 7f4ea88ae640 1 -- 172.21.15.156:0/3207312236 wait complete. 2024-03-08T08:28:50.039 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new cd571d9f-d56f-4013-95ef-1d1b69a3a852 4 2024-03-08T08:28:50.043 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.041+0000 7fb305639040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T08:28:50.090 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.088+0000 7fb305639040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T08:28:50.096 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.095+0000 7fc5dc847040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T08:28:50.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.143+0000 7f1a2df81640 1 Processor -- start 2024-03-08T08:28:50.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.143+0000 7f1a2df81640 1 -- start start 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.143+0000 7f1a2df81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a2815e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.143+0000 7f1a2df81640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a280c5c80 0x7f1a280c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a2df81640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280c7250 con 0x7f1a2815ae70 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a2df81640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2815c490 con 0x7f1a280c5c80 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a2df81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815eea0 con 0x7f1a2815e580 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a277fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a2815e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a27fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1a2815ae70 0x7f1a2815b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44032/0 (socket says 172.21.15.156:44032) 2024-03-08T08:28:50.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a27fff640 1 -- 172.21.15.156:0/3656019020 learned_addr learned my addr 172.21.15.156:0/3656019020 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a26ffd640 1 --2- 172.21.15.156:0/3656019020 >> v2:172.21.15.12:3300/0 conn(0x7f1a280c5c80 0x7f1a280c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a277fe640 1 -- 172.21.15.156:0/3656019020 >> v2:172.21.15.12:3300/0 conn(0x7f1a280c5c80 msgr2=0x7f1a280c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a277fe640 1 --2- 172.21.15.156:0/3656019020 >> v2:172.21.15.12:3300/0 conn(0x7f1a280c5c80 0x7f1a280c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a277fe640 1 -- 172.21.15.156:0/3656019020 >> v1:172.21.15.156:6789/0 conn(0x7f1a2815ae70 legacy=0x7f1a2815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a267fc640 1 -- 172.21.15.156:0/3656019020 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305992554 0 0) 0x7f1a280c7250 con 0x7f1a2815ae70 2024-03-08T08:28:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.144+0000 7f1a277fe640 1 -- 172.21.15.156:0/3656019020 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280c69b0 con 0x7f1a2815e580 2024-03-08T08:28:50.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a277fe640 1 --2- 172.21.15.156:0/3656019020 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a2815e960 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1a180097a0 tx=0x7f1a1802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5354f02a8f13b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a267fc640 1 -- 172.21.15.156:0/3656019020 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180315a0 con 0x7f1a2815e580 2024-03-08T08:28:50.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a267fc640 1 -- 172.21.15.156:0/3656019020 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a18037070 con 0x7f1a2815e580 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a267fc640 1 -- 172.21.15.156:0/3656019020 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18031960 con 0x7f1a2815e580 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 -- 172.21.15.156:0/3656019020 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 msgr2=0x7f1a2815e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 --2- 172.21.15.156:0/3656019020 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a2815e960 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1a180097a0 tx=0x7f1a1802b840 comp rx=0 tx=0).stop 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 -- 172.21.15.156:0/3656019020 shutdown_connections 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 --2- 172.21.15.156:0/3656019020 >> v2:172.21.15.12:3300/0 conn(0x7f1a280c5c80 0x7f1a280c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 --2- 172.21.15.156:0/3656019020 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a2815e960 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 -- 172.21.15.156:0/3656019020 >> 172.21.15.156:0/3656019020 conn(0x7f1a28099a40 msgr2=0x7f1a28099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 -- 172.21.15.156:0/3656019020 shutdown_connections 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.145+0000 7f1a2df81640 1 -- 172.21.15.156:0/3656019020 wait complete. 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 Processor -- start 2024-03-08T08:28:50.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 -- start start 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2815ae70 0x7f1a280ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a280adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280c6620 con 0x7f1a2815ae70 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280c6f70 con 0x7f1a280c5c80 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.147+0000 7f1a2df81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280b0060 con 0x7f1a2815e580 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a27fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a280adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50116/0 (socket says 172.21.15.156:50116) 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 -- 172.21.15.156:0/101058399 learned_addr learned my addr 172.21.15.156:0/101058399 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a26ffd640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2815ae70 0x7f1a280ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 -- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 msgr2=0x7f1a280adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a280adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 -- 172.21.15.156:0/101058399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2815ae70 msgr2=0x7f1a280ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2815ae70 0x7f1a280ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 -- 172.21.15.156:0/101058399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280b0430 con 0x7f1a280c5c80 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a27fff640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a280adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a277fe640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a18034040 tx=0x7f1a180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18032630 con 0x7f1a280c5c80 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180327d0 con 0x7f1a280c5c80 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18032bc0 con 0x7f1a280c5c80 2024-03-08T08:28:50.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a280b0710 con 0x7f1a280c5c80 2024-03-08T08:28:50.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a280b0cd0 con 0x7f1a280c5c80 2024-03-08T08:28:50.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.148+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1a18006020 con 0x7f1a280c5c80 2024-03-08T08:28:50.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.149+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f1a1802f030 con 0x7f1a280c5c80 2024-03-08T08:28:50.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.149+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a280749c0 con 0x7f1a280c5c80 2024-03-08T08:28:50.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.151+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1a18040070 con 0x7f1a280c5c80 2024-03-08T08:28:50.169 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.168+0000 7fc5dc847040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T08:28:50.211 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.209+0000 7fb305639040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T08:28:50.217 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.216+0000 7fc5dc847040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T08:28:50.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.223+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cd571d9f-d56f-4013-95ef-1d1b69a3a852", "id": 4} v 0) v1 -- 0x7f1a280c57c0 con 0x7f1a280c5c80 2024-03-08T08:28:50.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.228+0000 7f1a0ffff640 1 -- 172.21.15.156:0/101058399 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd571d9f-d56f-4013-95ef-1d1b69a3a852", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1a18032d60 con 0x7f1a280c5c80 2024-03-08T08:28:50.230 INFO:teuthology.orchestra.run.smithi156.stdout:4 2024-03-08T08:28:50.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 msgr2=0x7f1a28151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a18034040 tx=0x7f1a180046c0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 shutdown_connections 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1a2815e580 0x7f1a280adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2815ae70 0x7f1a280ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 --2- 172.21.15.156:0/101058399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a280c5c80 0x7f1a28151280 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 >> 172.21.15.156:0/101058399 conn(0x7f1a28099a40 msgr2=0x7f1a281597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 shutdown_connections 2024-03-08T08:28:50.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.229+0000 7f1a2df81640 1 -- 172.21.15.156:0/101058399 wait complete. 2024-03-08T08:28:50.242 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 45e09db2-d6a1-46dd-b7c1-4436bd779079 5 2024-03-08T08:28:50.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.270+0000 7fc5dc847040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T08:28:50.285 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.284+0000 7fb305639040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T08:28:50.320 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.319+0000 7fc5dc847040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T08:28:50.334 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.332+0000 7fb305639040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T08:28:50.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.341+0000 7fe8a5199640 1 Processor -- start 2024-03-08T08:28:50.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.341+0000 7fe8a5199640 1 -- start start 2024-03-08T08:28:50.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe8a5199640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8a00cdb10 0x7fe8a006f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe8a5199640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe8a5199640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a006e6a0 con 0x7fe8a006fd00 2024-03-08T08:28:50.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe8a5199640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a00d1650 con 0x7fe8a00cdb10 2024-03-08T08:28:50.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe8a5199640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a006e980 con 0x7fe8a00afbc0 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:42016/0 (socket says 172.21.15.156:42016) 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 -- 172.21.15.156:0/1795203493 learned_addr learned my addr 172.21.15.156:0/1795203493 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89e59c640 1 --2- 172.21.15.156:0/1795203493 >> v2:172.21.15.12:3300/0 conn(0x7fe8a00cdb10 0x7fe8a006f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 -- 172.21.15.156:0/1795203493 >> v2:172.21.15.12:3300/0 conn(0x7fe8a00cdb10 msgr2=0x7fe8a006f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 --2- 172.21.15.156:0/1795203493 >> v2:172.21.15.12:3300/0 conn(0x7fe8a00cdb10 0x7fe8a006f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 -- 172.21.15.156:0/1795203493 >> v1:172.21.15.12:6789/0 conn(0x7fe8a006fd00 legacy=0x7fe8a00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.342+0000 7fe89f59e640 1 -- 172.21.15.156:0/1795203493 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a006ec60 con 0x7fe8a00afbc0 2024-03-08T08:28:50.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe89f59e640 1 --2- 172.21.15.156:0/1795203493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe898009e50 tx=0x7fe89802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=345e8a51841c6913 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe89dd9b640 1 -- 172.21.15.156:0/1795203493 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89802cb00 con 0x7fe8a00afbc0 2024-03-08T08:28:50.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe89dd9b640 1 -- 172.21.15.156:0/1795203493 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89802cca0 con 0x7fe8a00afbc0 2024-03-08T08:28:50.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe89dd9b640 1 -- 172.21.15.156:0/1795203493 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898034610 con 0x7fe8a00afbc0 2024-03-08T08:28:50.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe8a5199640 1 -- 172.21.15.156:0/1795203493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 msgr2=0x7fe8a00cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.343+0000 7fe8a5199640 1 --2- 172.21.15.156:0/1795203493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe898009e50 tx=0x7fe89802bf60 comp rx=0 tx=0).stop 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 -- 172.21.15.156:0/1795203493 shutdown_connections 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 --2- 172.21.15.156:0/1795203493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00afbc0 0x7fe8a00cae30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 --2- 172.21.15.156:0/1795203493 >> v2:172.21.15.12:3300/0 conn(0x7fe8a00cdb10 0x7fe8a006f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 -- 172.21.15.156:0/1795203493 >> 172.21.15.156:0/1795203493 conn(0x7fe8a0099a40 msgr2=0x7fe8a0099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 -- 172.21.15.156:0/1795203493 shutdown_connections 2024-03-08T08:28:50.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.344+0000 7fe8a5199640 1 -- 172.21.15.156:0/1795203493 wait complete. 2024-03-08T08:28:50.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.345+0000 7fe8a5199640 1 Processor -- start 2024-03-08T08:28:50.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.345+0000 7fe8a5199640 1 -- start start 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 0x7fe8a00c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8a00afbc0 0x7fe8a00b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a00b0050 con 0x7fe8a00afbc0 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a006f220 con 0x7fe8a006fd00 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe8a5199640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a00b01d0 con 0x7fe8a00cdb10 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:42026/0 (socket says 172.21.15.156:42026) 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 -- 172.21.15.156:0/3006121154 learned_addr learned my addr 172.21.15.156:0/3006121154 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 0x7fe8a00c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89e59c640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8a00afbc0 0x7fe8a00b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 -- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 msgr2=0x7fe8a00c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 0x7fe8a00c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 -- 172.21.15.156:0/3006121154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8a00afbc0 msgr2=0x7fe8a00b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8a00afbc0 0x7fe8a00b6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 -- 172.21.15.156:0/3006121154 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a015ce20 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89ed9d640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 0x7fe8a00c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.346+0000 7fe89f59e640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe898036040 tx=0x7fe898004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898043020 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89802ce50 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a00ca860 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898009510 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a015d3a0 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe898046020 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.347+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fe89803ed00 con 0x7fe8a00cdb10 2024-03-08T08:28:50.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.348+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a00749c0 con 0x7fe8a00cdb10 2024-03-08T08:28:50.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.351+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe89804d030 con 0x7fe8a00cdb10 2024-03-08T08:28:50.391 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.390+0000 7fb305639040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T08:28:50.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.424+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "45e09db2-d6a1-46dd-b7c1-4436bd779079", "id": 5} v 0) v1 -- 0x7fe8a00caba0 con 0x7fe8a00cdb10 2024-03-08T08:28:50.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.429+0000 7fe8877fe640 1 -- 172.21.15.156:0/3006121154 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45e09db2-d6a1-46dd-b7c1-4436bd779079", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe89802cb10 con 0x7fe8a00cdb10 2024-03-08T08:28:50.431 INFO:teuthology.orchestra.run.smithi156.stdout:5 2024-03-08T08:28:50.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 msgr2=0x7fe8a00ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe898036040 tx=0x7fe898004a40 comp rx=0 tx=0).stop 2024-03-08T08:28:50.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 shutdown_connections 2024-03-08T08:28:50.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8a00cdb10 0x7fe8a00ca000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8a00afbc0 0x7fe8a00b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 --2- 172.21.15.156:0/3006121154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8a006fd00 0x7fe8a00c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 >> 172.21.15.156:0/3006121154 conn(0x7fe8a0099a40 msgr2=0x7fe8a00cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 shutdown_connections 2024-03-08T08:28:50.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.430+0000 7fe8a5199640 1 -- 172.21.15.156:0/3006121154 wait complete. 2024-03-08T08:28:50.441 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.440+0000 7fb305639040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T08:28:50.442 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 01b92c46-4426-4ab9-b1d2-d385d65ebdc3 6 2024-03-08T08:28:50.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.541+0000 7f9bd4d2e640 1 Processor -- start 2024-03-08T08:28:50.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.541+0000 7f9bd4d2e640 1 -- start start 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9bd015dc40 0x7f9bd00cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 0x7f9bd0099c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd006c030 con 0x7f9bd00d1320 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd006d110 con 0x7f9bd015dc40 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bd4d2e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00d0280 con 0x7f9bd015e020 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bce59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9bd015dc40 0x7f9bd00cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:42042/0 (socket says 172.21.15.156:42042) 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bce59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9bd015dc40 0x7f9bd00cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50138/0 (socket says 172.21.15.156:50138) 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bc7fff640 1 -- 172.21.15.156:0/3490081088 learned_addr learned my addr 172.21.15.156:0/3490081088 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.542+0000 7f9bced9d640 1 --2- 172.21.15.156:0/3490081088 >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 0x7f9bd0099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 -- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 msgr2=0x7f9bd006cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 -- 172.21.15.156:0/3490081088 >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 msgr2=0x7f9bd0099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 --2- 172.21.15.156:0/3490081088 >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 0x7f9bd0099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 -- 172.21.15.156:0/3490081088 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00d0560 con 0x7f9bd015dc40 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bc7fff640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bced9d640 1 --2- 172.21.15.156:0/3490081088 >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 0x7f9bd0099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:50.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bce59c640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00cfbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0072d00 tx=0x7f9bc002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=801bb767b6df495e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.543+0000 7f9bcdd9b640 1 -- 172.21.15.156:0/3490081088 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00315a0 con 0x7f9bd015dc40 2024-03-08T08:28:50.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bcdd9b640 1 -- 172.21.15.156:0/3490081088 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc0037070 con 0x7f9bd015dc40 2024-03-08T08:28:50.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bcdd9b640 1 -- 172.21.15.156:0/3490081088 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0031960 con 0x7f9bd015dc40 2024-03-08T08:28:50.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 msgr2=0x7f9bd00cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00cfbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0072d00 tx=0x7f9bc002b840 comp rx=0 tx=0).stop 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3490081088 shutdown_connections 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3490081088 >> v2:172.21.15.12:3300/0 conn(0x7f9bd00d1320 0x7f9bd0099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd015e020 0x7f9bd006cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3490081088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00cfbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3490081088 >> 172.21.15.156:0/3490081088 conn(0x7f9bd00b3490 msgr2=0x7f9bd00b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3490081088 shutdown_connections 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.544+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3490081088 wait complete. 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.545+0000 7f9bd4d2e640 1 Processor -- start 2024-03-08T08:28:50.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.545+0000 7f9bd4d2e640 1 -- start start 2024-03-08T08:28:50.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd00c7160 0x7f9bd00c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00d0e00 con 0x7f9bd00c7160 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00d0b20 con 0x7f9bd015dc40 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bd4d2e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd009a140 con 0x7f9bd00d1320 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bce59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bce59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:42058/0 (socket says 172.21.15.156:42058) 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50140/0 (socket says 172.21.15.156:50140) 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bce59c640 1 -- 172.21.15.156:0/3641691734 learned_addr learned my addr 172.21.15.156:0/3641691734 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.546+0000 7f9bced9d640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd00c7160 0x7f9bd00c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 -- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 msgr2=0x7f9bd01f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 -- 172.21.15.156:0/3641691734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd00c7160 msgr2=0x7f9bd00c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd00c7160 0x7f9bd00c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 -- 172.21.15.156:0/3641691734 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00b0640 con 0x7f9bd015dc40 2024-03-08T08:28:50.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bce59c640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc7fff640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800e860 tx=0x7f9bb800ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8004030 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb80041d0 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd01f7d40 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb800c940 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd01f7780 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9bb8005990 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.547+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f9bb8006610 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.548+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd00749c0 con 0x7f9bd015dc40 2024-03-08T08:28:50.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.550+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9bb8017020 con 0x7f9bd015dc40 2024-03-08T08:28:50.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.623+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "01b92c46-4426-4ab9-b1d2-d385d65ebdc3", "id": 6} v 0) v1 -- 0x7f9bd00b7920 con 0x7f9bd015dc40 2024-03-08T08:28:50.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.628+0000 7f9bc6ffd640 1 -- 172.21.15.156:0/3641691734 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "01b92c46-4426-4ab9-b1d2-d385d65ebdc3", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9bb80093b0 con 0x7f9bd015dc40 2024-03-08T08:28:50.630 INFO:teuthology.orchestra.run.smithi156.stdout:6 2024-03-08T08:28:50.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 msgr2=0x7f9bd00b0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800e860 tx=0x7f9bb800ed20 comp rx=0 tx=0).stop 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 shutdown_connections 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd00c7160 0x7f9bd00c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd015dc40 0x7f9bd00b0100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 --2- 172.21.15.156:0/3641691734 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bd00d1320 0x7f9bd01f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 >> 172.21.15.156:0/3641691734 conn(0x7f9bd00b3490 msgr2=0x7f9bd0074130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 shutdown_connections 2024-03-08T08:28:50.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.629+0000 7f9bd4d2e640 1 -- 172.21.15.156:0/3641691734 wait complete. 2024-03-08T08:28:50.641 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 3ddbf98e-92c1-4352-8ebf-f1447907b04f 7 2024-03-08T08:28:50.713 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.712+0000 7fc5dc847040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T08:28:50.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.741+0000 7f2e393b5640 1 Processor -- start 2024-03-08T08:28:50.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 -- start start 2024-03-08T08:28:50.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e3415bb90 0x7f2e340cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e340d0340 0x7f2e340cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3409a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340cd590 con 0x7f2e340d0340 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340cdf60 con 0x7f2e3415bb90 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e393b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340cf700 con 0x7f2e34099b90 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e32ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e3415bb90 0x7f2e340cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e32ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e3415bb90 0x7f2e340cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50154/0 (socket says 172.21.15.156:50154) 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e32ffd640 1 -- 172.21.15.156:0/1874746986 learned_addr learned my addr 172.21.15.156:0/1874746986 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.742+0000 7f2e337fe640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3409a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e327fc640 1 --2- 172.21.15.156:0/1874746986 >> v2:172.21.15.12:3300/0 conn(0x7f2e340d0340 0x7f2e340cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e32ffd640 1 -- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 msgr2=0x7f2e3409a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e32ffd640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3409a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e32ffd640 1 -- 172.21.15.156:0/1874746986 >> v2:172.21.15.12:3300/0 conn(0x7f2e340d0340 msgr2=0x7f2e340cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e32ffd640 1 --2- 172.21.15.156:0/1874746986 >> v2:172.21.15.12:3300/0 conn(0x7f2e340d0340 0x7f2e340cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e32ffd640 1 -- 172.21.15.156:0/1874746986 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340cc980 con 0x7f2e3415bb90 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.743+0000 7f2e337fe640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3409a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e32ffd640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e3415bb90 0x7f2e340cc390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e34072080 tx=0x7f2e2802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bc2e6b72268b0cd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e31ffb640 1 -- 172.21.15.156:0/1874746986 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28035020 con 0x7f2e3415bb90 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e31ffb640 1 -- 172.21.15.156:0/1874746986 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28031870 con 0x7f2e3415bb90 2024-03-08T08:28:50.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e31ffb640 1 -- 172.21.15.156:0/1874746986 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28031c40 con 0x7f2e3415bb90 2024-03-08T08:28:50.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 -- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e3415bb90 msgr2=0x7f2e340cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e3415bb90 0x7f2e340cc390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e34072080 tx=0x7f2e2802b910 comp rx=0 tx=0).stop 2024-03-08T08:28:50.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 -- 172.21.15.156:0/1874746986 shutdown_connections 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3409a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 --2- 172.21.15.156:0/1874746986 >> v2:172.21.15.12:3300/0 conn(0x7f2e340d0340 0x7f2e340cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 --2- 172.21.15.156:0/1874746986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e3415bb90 0x7f2e340cc390 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 -- 172.21.15.156:0/1874746986 >> 172.21.15.156:0/1874746986 conn(0x7f2e340b3370 msgr2=0x7f2e340b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 -- 172.21.15.156:0/1874746986 shutdown_connections 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.744+0000 7f2e393b5640 1 -- 172.21.15.156:0/1874746986 wait complete. 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 Processor -- start 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 -- start start 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3414dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e3415bb90 0x7f2e340cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340cd220 con 0x7f2e3415bb90 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340ccf40 con 0x7f2e340d0340 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.746+0000 7f2e393b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340d0780 con 0x7f2e34099b90 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e32ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3414dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e32ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3414dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:42068/0 (socket says 172.21.15.156:42068) 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50168/0 (socket says 172.21.15.156:50168) 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e32ffd640 1 -- 172.21.15.156:0/256129015 learned_addr learned my addr 172.21.15.156:0/256129015 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e337fe640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e3415bb90 0x7f2e340cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 -- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 msgr2=0x7f2e3414dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3414dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 -- 172.21.15.156:0/256129015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e3415bb90 msgr2=0x7f2e340cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e3415bb90 0x7f2e340cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 -- 172.21.15.156:0/256129015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e341f7430 con 0x7f2e340d0340 2024-03-08T08:28:50.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e327fc640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e880 tx=0x7f2e2400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24017070 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2400c940 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e341f7710 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400cd30 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e341f7cd0 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2e2401a070 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.747+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f2e24013ad0 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.748+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e340749c0 con 0x7f2e340d0340 2024-03-08T08:28:50.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.750+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2e24013470 con 0x7f2e340d0340 2024-03-08T08:28:50.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.822+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3ddbf98e-92c1-4352-8ebf-f1447907b04f", "id": 7} v 0) v1 -- 0x7f2e340c7f50 con 0x7f2e340d0340 2024-03-08T08:28:50.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.825+0000 7f2e1b7fe640 1 -- 172.21.15.156:0/256129015 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ddbf98e-92c1-4352-8ebf-f1447907b04f", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e24013650 con 0x7f2e340d0340 2024-03-08T08:28:50.827 INFO:teuthology.orchestra.run.smithi156.stdout:7 2024-03-08T08:28:50.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 msgr2=0x7f2e340cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e880 tx=0x7f2e2400ec80 comp rx=0 tx=0).stop 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 shutdown_connections 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e3415bb90 0x7f2e340cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e340d0340 0x7f2e340cab20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 --2- 172.21.15.156:0/256129015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e34099b90 0x7f2e3414dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 >> 172.21.15.156:0/256129015 conn(0x7f2e340b3370 msgr2=0x7f2e34159ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 shutdown_connections 2024-03-08T08:28:50.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:28:50.826+0000 7f2e393b5640 1 -- 172.21.15.156:0/256129015 wait complete. 2024-03-08T08:28:50.839 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-08T08:28:50.839 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-08T08:28:50.841 INFO:tasks.ceph.osd.0:Started 2024-03-08T08:28:50.841 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-08T08:28:50.842 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-08T08:28:50.844 INFO:tasks.ceph.osd.1:Started 2024-03-08T08:28:50.844 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-08T08:28:50.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-08T08:28:50.846 INFO:tasks.ceph.osd.2:Started 2024-03-08T08:28:50.846 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-08T08:28:50.846 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-08T08:28:50.847 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:50.845+0000 7fb305639040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T08:28:50.848 INFO:tasks.ceph.osd.3:Started 2024-03-08T08:28:50.849 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-08T08:28:50.849 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-08T08:28:50.851 INFO:tasks.ceph.osd.4:Started 2024-03-08T08:28:50.851 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-08T08:28:50.851 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-08T08:28:50.853 INFO:tasks.ceph.osd.5:Started 2024-03-08T08:28:50.854 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-08T08:28:50.854 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-08T08:28:50.856 INFO:tasks.ceph.osd.6:Started 2024-03-08T08:28:50.856 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-08T08:28:50.856 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-08T08:28:50.857 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:50.856+0000 7fc5dc847040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T08:28:50.858 INFO:tasks.ceph.osd.7:Started 2024-03-08T08:28:50.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:28:50.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.979+0000 7f937a44e640 1 Processor -- start 2024-03-08T08:28:50.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.979+0000 7f937a44e640 1 -- start start 2024-03-08T08:28:50.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f937a44e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f93740d1110 0x7f93740b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f937a44e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93740b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f937a44e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937415e1a0 con 0x7f937415ddc0 2024-03-08T08:28:50.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f937a44e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93740b9030 con 0x7f93740d1110 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f937a44e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740d1d20 con 0x7f93740d31c0 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f9373fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f937415ddc0 0x7f93740d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45190/0 (socket says 172.21.15.12:45190) 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.980+0000 7f9373fff640 1 -- 172.21.15.12:0/2810801658 learned_addr learned my addr 172.21.15.12:0/2810801658 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93789eb640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93740b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93737fe640 1 --2- 172.21.15.12:0/2810801658 >> v2:172.21.15.156:3300/0 conn(0x7f93740d1110 0x7f93740b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f9372ffd640 1 -- 172.21.15.12:0/2810801658 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454408452 0 0) 0x7f937415e1a0 con 0x7f937415ddc0 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f9372ffd640 1 -- 172.21.15.12:0/2810801658 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003520 con 0x7f937415ddc0 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93737fe640 1 -- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 msgr2=0x7f93740b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93737fe640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93740b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93737fe640 1 -- 172.21.15.12:0/2810801658 >> v1:172.21.15.12:6789/0 conn(0x7f937415ddc0 legacy=0x7f93740d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93737fe640 1 -- 172.21.15.12:0/2810801658 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740b8790 con 0x7f93740d1110 2024-03-08T08:28:50.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.981+0000 7f93789eb640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93740b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f93737fe640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93740d1110 0x7f93740b6be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93640097a0 tx=0x7f936402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1186cd8d0c282db server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f9372ffd640 1 -- 172.21.15.12:0/2810801658 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364035020 con 0x7f93740d1110 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f9372ffd640 1 -- 172.21.15.12:0/2810801658 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364031810 con 0x7f93740d1110 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f9372ffd640 1 -- 172.21.15.12:0/2810801658 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031be0 con 0x7f93740d1110 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f937a44e640 1 -- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93740d1110 msgr2=0x7f93740b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f937a44e640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93740d1110 0x7f93740b6be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93640097a0 tx=0x7f936402b8b0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f937a44e640 1 -- 172.21.15.12:0/2810801658 shutdown_connections 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f937a44e640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93740b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.982+0000 7f937a44e640 1 --2- 172.21.15.12:0/2810801658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93740d1110 0x7f93740b6be0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.983+0000 7f937a44e640 1 -- 172.21.15.12:0/2810801658 >> 172.21.15.12:0/2810801658 conn(0x7f9374072720 msgr2=0x7f9374072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.983+0000 7f937a44e640 1 -- 172.21.15.12:0/2810801658 shutdown_connections 2024-03-08T08:28:50.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.983+0000 7f937a44e640 1 -- 172.21.15.12:0/2810801658 wait complete. 2024-03-08T08:28:50.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.984+0000 7f937a44e640 1 Processor -- start 2024-03-08T08:28:50.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.984+0000 7f937a44e640 1 -- start start 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93741f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f937415ddc0 0x7f9374153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b9980 con 0x7f93740d1110 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b8d50 con 0x7f937415ddc0 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b7660 con 0x7f93740d31c0 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47674/0 (socket says 172.21.15.12:47674) 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 -- 172.21.15.12:0/3697814544 learned_addr learned my addr 172.21.15.12:0/3697814544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f93789eb640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f937415ddc0 0x7f9374153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f93737fe640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93741f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:50.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 -- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 msgr2=0x7f93741f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93741f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 -- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f937415ddc0 msgr2=0x7f9374153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f937415ddc0 0x7f9374153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 -- 172.21.15.12:0/3697814544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93741f4630 con 0x7f93740d1110 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f93737fe640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93741f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9373fff640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f936800c3c0 tx=0x7f936800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368017070 con 0x7f93740d1110 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368005ce0 con 0x7f93740d1110 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93741f4bb0 con 0x7f93740d1110 2024-03-08T08:28:50.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680072f0 con 0x7f93740d1110 2024-03-08T08:28:50.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.985+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93741f5170 con 0x7f93740d1110 2024-03-08T08:28:50.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.986+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f936801a070 con 0x7f93740d1110 2024-03-08T08:28:50.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.986+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9368012ad0 con 0x7f93740d1110 2024-03-08T08:28:50.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.987+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93740749c0 con 0x7f93740d1110 2024-03-08T08:28:50.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:50.989+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9368021020 con 0x7f93740d1110 2024-03-08T08:28:51.008 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.007+0000 7fc5dc847040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T08:28:51.011 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.009+0000 7fb305639040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T08:28:51.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.051+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93740d3ac0 con 0x7f93740d1110 2024-03-08T08:28:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.052+0000 7f9370ff9640 1 -- 172.21.15.12:0/3697814544 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f93741f4630 con 0x7f93740d1110 2024-03-08T08:28:51.053 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:51.053 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:28:50.825188+0000","last_up_change":"0.000000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.052+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 msgr2=0x7f93740cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.052+0000 7f937a44e640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f936800c3c0 tx=0x7f936800c880 comp rx=0 tx=0).stop 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 shutdown_connections 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f937415ddc0 0x7f9374153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93740d31c0 0x7f93741f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 --2- 172.21.15.12:0/3697814544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93740d1110 0x7f93740cab80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 >> 172.21.15.12:0/3697814544 conn(0x7f9374072720 msgr2=0x7f93740d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 shutdown_connections 2024-03-08T08:28:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:51.053+0000 7f937a44e640 1 -- 172.21.15.12:0/3697814544 wait complete. 2024-03-08T08:28:51.055 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.055+0000 7fc5dc847040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T08:28:51.064 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-08T08:28:51.064 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-08T08:28:51.128 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.126+0000 7fb305639040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T08:28:51.172 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.171+0000 7fc5dc847040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T08:28:51.173 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.171+0000 7fb305639040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T08:28:51.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.270+0000 7fc5dc847040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T08:28:51.286 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.285+0000 7fb305639040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T08:28:51.381 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.380+0000 7fb305639040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T08:28:51.430 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.429+0000 7fc5dc847040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T08:28:51.451 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-03-08T08:28:51.449+0000 7f388a612640 -1 Falling back to public interface 2024-03-08T08:28:51.461 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-03-08T08:28:51.459+0000 7f09c5be1640 -1 Falling back to public interface 2024-03-08T08:28:51.466 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-08T08:28:51.465+0000 7f7c6d215640 -1 Falling back to public interface 2024-03-08T08:28:51.470 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-08T08:28:51.469+0000 7f71475e1640 -1 Falling back to public interface 2024-03-08T08:28:51.471 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-08T08:28:51.469+0000 7f81c7e1f640 -1 Falling back to public interface 2024-03-08T08:28:51.471 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-08T08:28:51.470+0000 7fab188d3640 -1 Falling back to public interface 2024-03-08T08:28:51.472 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-08T08:28:51.471+0000 7f7146590640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2024-03-08T08:28:51.476 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.476+0000 7fc5dc847040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T08:28:51.481 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-03-08T08:28:51.479+0000 7fc246610640 -1 Falling back to public interface 2024-03-08T08:28:51.485 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-03-08T08:28:51.483+0000 7fd381506640 -1 Falling back to public interface 2024-03-08T08:28:51.542 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.540+0000 7fb305639040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T08:28:51.584 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.582+0000 7fb305639040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T08:28:51.638 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:51.637+0000 7fc5dc847040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T08:28:51.740 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:51.739+0000 7fb305639040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T08:28:51.868 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-08T08:28:51.957 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.956+0000 7f36cfad4640 1 Processor -- start 2024-03-08T08:28:51.958 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 -- start start 2024-03-08T08:28:51.959 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f36c815dea0 0x7f36c80b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:51.959 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f36c80cfb70 0x7f36c80d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:51.959 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c815e280 con 0x7f36c80ce400 2024-03-08T08:28:51.959 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80b1f10 con 0x7f36c80cfb70 2024-03-08T08:28:51.959 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.957+0000 7f36cfad4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80ccbb0 con 0x7f36c815dea0 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36ce071640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f36c80cfb70 0x7f36c80d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36cd06f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36c80ce400 0x7f36c80cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45200/0 (socket says 172.21.15.12:45200) 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36cd06f640 1 -- 172.21.15.12:0/409927889 learned_addr learned my addr 172.21.15.12:0/409927889 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36cd870640 1 --2- 172.21.15.12:0/409927889 >> v2:172.21.15.156:3300/0 conn(0x7f36c815dea0 0x7f36c80b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36ce071640 1 -- 172.21.15.12:0/409927889 >> v2:172.21.15.156:3300/0 conn(0x7f36c815dea0 msgr2=0x7f36c80b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36ce071640 1 --2- 172.21.15.12:0/409927889 >> v2:172.21.15.156:3300/0 conn(0x7f36c815dea0 0x7f36c80b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36ce071640 1 -- 172.21.15.12:0/409927889 >> v1:172.21.15.12:6789/0 conn(0x7f36c80ce400 legacy=0x7f36c80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:51.960 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.958+0000 7f36ce071640 1 -- 172.21.15.12:0/409927889 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80cce90 con 0x7f36c80cfb70 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36ce071640 1 --2- 172.21.15.12:0/409927889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c80d0020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f36c8070e70 tx=0x7f36c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e04a6d0ceb5ee54 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cc86e640 1 -- 172.21.15.12:0/409927889 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0039070 con 0x7f36c80cfb70 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cc86e640 1 -- 172.21.15.12:0/409927889 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c002cbf0 con 0x7f36c80cfb70 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cc86e640 1 -- 172.21.15.12:0/409927889 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0034440 con 0x7f36c80cfb70 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 -- 172.21.15.12:0/409927889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 msgr2=0x7f36c80d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 --2- 172.21.15.12:0/409927889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c80d0020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f36c8070e70 tx=0x7f36c002be40 comp rx=0 tx=0).stop 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 -- 172.21.15.12:0/409927889 shutdown_connections 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 --2- 172.21.15.12:0/409927889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c80d0020 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 --2- 172.21.15.12:0/409927889 >> v2:172.21.15.156:3300/0 conn(0x7f36c815dea0 0x7f36c80b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.959+0000 7f36cfad4640 1 -- 172.21.15.12:0/409927889 >> 172.21.15.12:0/409927889 conn(0x7f36c80b2380 msgr2=0x7f36c80b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.960+0000 7f36cfad4640 1 -- 172.21.15.12:0/409927889 shutdown_connections 2024-03-08T08:28:51.962 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.960+0000 7f36cfad4640 1 -- 172.21.15.12:0/409927889 wait complete. 2024-03-08T08:28:51.963 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 Processor -- start 2024-03-08T08:28:51.963 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 -- start start 2024-03-08T08:28:51.965 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 0x7f36c81f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:51.965 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:51.965 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36c815dea0 0x7f36c80b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:51.965 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b1b70 con 0x7f36c80cfb70 2024-03-08T08:28:51.965 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80d0560 con 0x7f36c815dea0 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cfad4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b1cf0 con 0x7f36c80ce400 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cd06f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cd06f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47692/0 (socket says 172.21.15.12:47692) 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36cd06f640 1 -- 172.21.15.12:0/4056531115 learned_addr learned my addr 172.21.15.12:0/4056531115 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.961+0000 7f36ce071640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36c815dea0 0x7f36c80b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd870640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 0x7f36c81f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 -- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 msgr2=0x7f36c81f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 0x7f36c81f4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 -- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36c815dea0 msgr2=0x7f36c80b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36c815dea0 0x7f36c80b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:51.966 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 -- 172.21.15.12:0/4056531115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c81f5000 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd870640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 0x7f36c81f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cd06f640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c440 tx=0x7f36b8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8017070 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b8005ce0 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80072f0 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c81f5740 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c80ae120 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f36b801a070 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.962+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f36b8012ad0 con 0x7f36c80cfb70 2024-03-08T08:28:51.967 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.963+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c8074fe0 con 0x7f36c80cfb70 2024-03-08T08:28:51.968 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:51.965+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f36b8021020 con 0x7f36c80cfb70 2024-03-08T08:28:52.062 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.061+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c80b67b0 con 0x7f36c80cfb70 2024-03-08T08:28:52.062 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.061+0000 7f36be7fc640 1 -- 172.21.15.12:0/4056531115 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f36c81f5000 con 0x7f36c80cfb70 2024-03-08T08:28:52.063 INFO:teuthology.misc.health.smithi012.stdout: 2024-03-08T08:28:52.063 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":10,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:28:50.825188+0000","last_up_change":"0.000000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:28:52.065 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 msgr2=0x7f36c81f5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:52.065 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c440 tx=0x7f36b8001e70 comp rx=0 tx=0).stop 2024-03-08T08:28:52.065 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 shutdown_connections 2024-03-08T08:28:52.065 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36c815dea0 0x7f36c80b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:52.066 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c80cfb70 0x7f36c81f5200 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:52.066 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 --2- 172.21.15.12:0/4056531115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36c80ce400 0x7f36c81f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:52.066 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 >> 172.21.15.12:0/4056531115 conn(0x7f36c80b2380 msgr2=0x7f36c80c61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:52.066 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 shutdown_connections 2024-03-08T08:28:52.066 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:52.062+0000 7f36cfad4640 1 -- 172.21.15.12:0/4056531115 wait complete. 2024-03-08T08:28:52.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-08T08:28:52.515 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:52.514+0000 7fc5dc847040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T08:28:52.802 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:52.801+0000 7fc5dc847040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T08:28:52.833 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:52.831+0000 7fb305639040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T08:28:52.855 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:52.854+0000 7fc5dc847040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T08:28:52.974 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:52.973+0000 7fc5dc847040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T08:28:53.113 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:53.111+0000 7fb305639040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T08:28:53.166 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:53.164+0000 7fb305639040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T08:28:53.280 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-03-08T08:28:53.278+0000 7fb305639040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T08:28:53.998 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:53.997+0000 7fc57d426640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-08T08:28:53.999 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-08T08:28:53.997+0000 7fc57d426640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-08T08:28:54.903 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-03-08T08:28:54.901+0000 7f388a612640 -1 osd.6 0 log_to_monitors true 2024-03-08T08:28:54.954 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-03-08T08:28:54.952+0000 7f09c5be1640 -1 osd.7 0 log_to_monitors true 2024-03-08T08:28:54.957 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-08T08:28:54.956+0000 7f81c7e1f640 -1 osd.0 0 log_to_monitors true 2024-03-08T08:28:54.980 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-08T08:28:54.979+0000 7f7c6d215640 -1 osd.2 0 log_to_monitors true 2024-03-08T08:28:54.981 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-08T08:28:54.980+0000 7fab188d3640 -1 osd.3 0 log_to_monitors true 2024-03-08T08:28:54.989 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-03-08T08:28:54.987+0000 7fc246610640 -1 osd.4 0 log_to_monitors true 2024-03-08T08:28:54.992 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-08T08:28:54.991+0000 7f71475e1640 -1 osd.1 0 log_to_monitors true 2024-03-08T08:28:55.000 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-03-08T08:28:54.998+0000 7fd381506640 -1 osd.5 0 log_to_monitors true 2024-03-08T08:28:57.164 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-03-08T08:28:57.162+0000 7fc241736640 -1 osd.4 0 waiting for initial osdmap 2024-03-08T08:28:57.174 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-03-08T08:28:57.169+0000 7f09c1354640 -1 osd.7 0 waiting for initial osdmap 2024-03-08T08:28:57.174 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-03-08T08:28:57.171+0000 7fc239d68640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.181 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-03-08T08:28:57.179+0000 7f09b9986640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.183 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-03-08T08:28:57.181+0000 7f3886ff9640 -1 osd.6 0 waiting for initial osdmap 2024-03-08T08:28:57.197 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-03-08T08:28:57.196+0000 7f387ddf3640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.776 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-03-08T08:28:57.774+0000 7fd37cc79640 -1 osd.5 0 waiting for initial osdmap 2024-03-08T08:28:57.791 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-03-08T08:28:57.790+0000 7fd3752ab640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.932 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-08T08:28:57.931+0000 7fab14046640 -1 osd.3 0 waiting for initial osdmap 2024-03-08T08:28:57.949 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-08T08:28:57.948+0000 7fab0c678640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.971 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-08T08:28:57.969+0000 7f7c6845a640 -1 osd.2 0 waiting for initial osdmap 2024-03-08T08:28:57.977 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-08T08:28:57.976+0000 7f81c2fd4640 -1 osd.0 0 waiting for initial osdmap 2024-03-08T08:28:57.984 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-08T08:28:57.983+0000 7f7c60a8c640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:57.994 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-08T08:28:57.994+0000 7f81bb606640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:58.128 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-08T08:28:58.127+0000 7f7142d54640 -1 osd.1 0 waiting for initial osdmap 2024-03-08T08:28:58.140 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-08T08:28:58.139+0000 7f713b386640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T08:28:58.879 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-08T08:28:58.980 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.979+0000 7fd60e833640 1 Processor -- start 2024-03-08T08:28:58.981 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.979+0000 7fd60e833640 1 -- start start 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60e833640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd6080b2c60 0x7fd6080b9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60e833640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6080b9a60 con 0x7fd6080b6b70 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60e833640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6080d1f50 con 0x7fd60806c250 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60e833640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6080d1800 con 0x7fd6080b2c60 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60cdd0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6080b6b70 0x7fd6080d2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44920/0 (socket says 172.21.15.12:44920) 2024-03-08T08:28:58.984 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd60cdd0640 1 -- 172.21.15.12:0/546029975 learned_addr learned my addr 172.21.15.12:0/546029975 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd6077fe640 1 --2- 172.21.15.12:0/546029975 >> v2:172.21.15.156:3300/0 conn(0x7fd6080b2c60 0x7fd6080b9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.980+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584755011 0 0) 0x7fd6080b9a60 con 0x7fd6080b6b70 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f4003520 con 0x7fd6080b6b70 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186510770 0 0) 0x7fd5f4003520 con 0x7fd6080b6b70 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 >> v2:172.21.15.156:3300/0 conn(0x7fd6080b2c60 msgr2=0x7fd6080b9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 --2- 172.21.15.12:0/546029975 >> v2:172.21.15.156:3300/0 conn(0x7fd6080b2c60 0x7fd6080b9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 >> v1:172.21.15.156:6789/0 conn(0x7fd60806c250 legacy=0x7fd60806c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080d2230 con 0x7fd6080b6b70 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fd6000019f0 con 0x7fd6080b6b70 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 >> v1:172.21.15.12:6789/0 conn(0x7fd6080b6b70 legacy=0x7fd6080d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f40039b0 0x7fd5f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f4004800 0x7fd5f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 0x7fd5f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400c040 con 0x7fd5f40051c0 2024-03-08T08:28:58.985 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400c1f0 con 0x7fd5f40039b0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400c3f0 con 0x7fd5f4004800 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 0x7fd5f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd607fff640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f40039b0 0x7fd5f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd6077fe640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f4004800 0x7fd5f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 -- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f4004800 msgr2=0x7fd5f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f4004800 0x7fd5f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 -- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f40039b0 msgr2=0x7fd5f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f40039b0 0x7fd5f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.981+0000 7fd60cdd0640 1 -- 172.21.15.12:0/546029975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f4003520 con 0x7fd5f40051c0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.982+0000 7fd60cdd0640 1 -- 172.21.15.12:0/546029975 reap_dead start 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.982+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 0x7fd5f400b9c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd60000a840 tx=0x7fd5f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707e2b4814a01672 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.982+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60000a460 con 0x7fd5f40051c0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.982+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd600034650 con 0x7fd5f40051c0 2024-03-08T08:28:58.986 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.982+0000 7fd606ffd640 1 -- 172.21.15.12:0/546029975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600034a40 con 0x7fd5f40051c0 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 -- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 msgr2=0x7fd5f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 0x7fd5f400b9c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd60000a840 tx=0x7fd5f4005830 comp rx=0 tx=0).stop 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 -- 172.21.15.12:0/546029975 shutdown_connections 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 --2- 172.21.15.12:0/546029975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f40051c0 0x7fd5f400b9c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 -- 172.21.15.12:0/546029975 >> 172.21.15.12:0/546029975 conn(0x7fd6080727a0 msgr2=0x7fd608072bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 -- 172.21.15.12:0/546029975 shutdown_connections 2024-03-08T08:28:58.987 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.983+0000 7fd60e833640 1 -- 172.21.15.12:0/546029975 wait complete. 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.984+0000 7fd60e833640 1 Processor -- start 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.984+0000 7fd60e833640 1 -- start start 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 0x7fd60814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 0x7fd6080b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6080b7000 con 0x7fd6080b61a0 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6080d2880 con 0x7fd6080b5d40 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60e833640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6080b7180 con 0x7fd5f40051c0 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53748/0 (socket says 172.21.15.12:53748) 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 -- 172.21.15.12:0/1357140347 learned_addr learned my addr 172.21.15.12:0/1357140347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd6077fe640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 0x7fd6080b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd607fff640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 0x7fd60814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.990 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 -- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 msgr2=0x7fd60814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 0x7fd60814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 -- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 msgr2=0x7fd6080b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 0x7fd6080b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd60cdd0640 1 -- 172.21.15.12:0/1357140347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60814d500 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd607fff640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 0x7fd60814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.985+0000 7fd6077fe640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 0x7fd6080b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd60cdd0640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd60002cc50 tx=0x7fd6080b6ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600044070 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd600004de0 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd608158e00 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600039040 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.986+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6081593c0 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.987+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608074fe0 con 0x7fd6080b61a0 2024-03-08T08:28:58.991 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.987+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fd600047070 con 0x7fd6080b61a0 2024-03-08T08:28:58.992 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.988+0000 7fd604ff9640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 0x7fd5e806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:58.992 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.988+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4191+0+0 (secure 0 0 0) 0x7fd6000ab230 con 0x7fd6080b61a0 2024-03-08T08:28:58.992 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.988+0000 7fd607fff640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 0x7fd5e806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:58.992 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.988+0000 7fd607fff640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 0x7fd5e806d900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c00 tx=0x7fd5f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:58.993 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:58.992+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd60007a130 con 0x7fd6080b61a0 2024-03-08T08:28:59.182 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.181+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6080d12d0 con 0x7fd6080b61a0 2024-03-08T08:28:59.183 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.182+0000 7fd604ff9640 1 -- 172.21.15.12:0/1357140347 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12865 (secure 0 0 0) 0x7fd60007e1d0 con 0x7fd6080b61a0 2024-03-08T08:28:59.183 INFO:teuthology.misc.health.smithi012.stdout: 2024-03-08T08:28:59.183 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":15,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:28:59.148406+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 msgr2=0x7fd5e806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 0x7fd5e806d900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c00 tx=0x7fd5f0009040 comp rx=0 tx=0).stop 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 msgr2=0x7fd6080b3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd60002cc50 tx=0x7fd6080b6ac0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 shutdown_connections 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd5e806b440 0x7fd5e806d900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.184+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6080b61a0 0x7fd6080b3cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.185+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6080b5d40 0x7fd6080b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.186 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.185+0000 7fd60e833640 1 --2- 172.21.15.12:0/1357140347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5f40051c0 0x7fd60814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.187 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.185+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 >> 172.21.15.12:0/1357140347 conn(0x7fd6080727a0 msgr2=0x7fd60815f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.187 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.185+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 shutdown_connections 2024-03-08T08:28:59.187 INFO:teuthology.misc.health.smithi012.stderr:2024-03-08T08:28:59.185+0000 7fd60e833640 1 -- 172.21.15.12:0/1357140347 wait complete. 2024-03-08T08:28:59.195 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-08T08:28:59.195 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-08T08:28:59.196 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-08T08:28:59.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-08T08:28:59.198 INFO:tasks.ceph.mds.a:Started 2024-03-08T08:28:59.198 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-08T08:28:59.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-08T08:28:59.201 INFO:tasks.ceph.mds.c:Started 2024-03-08T08:28:59.201 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-08T08:28:59.201 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-08T08:28:59.204 INFO:tasks.ceph.mds.b:Started 2024-03-08T08:28:59.204 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-08T08:28:59.204 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-08T08:28:59.206 INFO:tasks.ceph.mds.d:Started 2024-03-08T08:28:59.206 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-08T08:28:59.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:28:59.275 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-08T08:28:59.275 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-08T08:28:59.290 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2024-03-08T08:28:59.290 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2024-03-08T08:28:59.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.310+0000 7fae10cff640 1 Processor -- start 2024-03-08T08:28:59.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.311+0000 7fae10cff640 1 -- start start 2024-03-08T08:28:59.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.311+0000 7fae10cff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 0x7fae0c0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.311+0000 7fae10cff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fae0c0cd950 0x7fae0c06fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.311+0000 7fae10cff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c0b31e0 con 0x7fae0c070130 2024-03-08T08:28:59.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.311+0000 7fae10cff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0c0b34c0 con 0x7fae0c0cd950 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae10cff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c0b37a0 con 0x7fae0c15a8b0 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fae0c0cd950 0x7fae0c06fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae0a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 0x7fae0c0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fae0c0cd950 0x7fae0c06fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:33890/0 (socket says 172.21.15.12:33890) 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae0ad9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fae0c070130 0x7fae0c15e4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:57410/0 (socket says 172.21.15.12:57410) 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 -- 172.21.15.12:0/1063658786 learned_addr learned my addr 172.21.15.12:0/1063658786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 -- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 msgr2=0x7fae0c0cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 0x7fae0c0cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 -- 172.21.15.12:0/1063658786 >> v1:172.21.15.156:6789/0 conn(0x7fae0c070130 legacy=0x7fae0c15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae09d9b640 1 -- 172.21.15.12:0/1063658786 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c0b3a80 con 0x7fae0c0cd950 2024-03-08T08:28:59.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.312+0000 7fae0a59c640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 0x7fae0c0cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:59.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae09d9b640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c0cd950 0x7fae0c06fbf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009870 tx=0x7fadfc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fd0cbaa4ac76ef server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae0959a640 1 -- 172.21.15.12:0/1063658786 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc035020 con 0x7fae0c0cd950 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae0959a640 1 -- 172.21.15.12:0/1063658786 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc031640 con 0x7fae0c0cd950 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae0959a640 1 -- 172.21.15.12:0/1063658786 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc02d040 con 0x7fae0c0cd950 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae10cff640 1 -- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c0cd950 msgr2=0x7fae0c06fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.313+0000 7fae10cff640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c0cd950 0x7fae0c06fbf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009870 tx=0x7fadfc02b8f0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 -- 172.21.15.12:0/1063658786 shutdown_connections 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c0cd950 0x7fae0c06fbf0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 --2- 172.21.15.12:0/1063658786 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c15a8b0 0x7fae0c0cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 -- 172.21.15.12:0/1063658786 >> 172.21.15.12:0/1063658786 conn(0x7fae0c06f0d0 msgr2=0x7fae0c06f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 -- 172.21.15.12:0/1063658786 shutdown_connections 2024-03-08T08:28:59.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.314+0000 7fae10cff640 1 -- 172.21.15.12:0/1063658786 wait complete. 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.315+0000 7fae10cff640 1 Processor -- start 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.315+0000 7fae10cff640 1 -- start start 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c070130 0x7fae0c151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c15a8b0 0x7fae0c147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c06bdc0 con 0x7fae0c0cd950 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c0cddf0 con 0x7fae0c15a8b0 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c06bf40 con 0x7fae0c070130 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53794/0 (socket says 172.21.15.12:53794) 2024-03-08T08:28:59.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 -- 172.21.15.12:0/4129785623 learned_addr learned my addr 172.21.15.12:0/4129785623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae0a59c640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c070130 0x7fae0c151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae0ad9d640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c15a8b0 0x7fae0c147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 -- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c070130 msgr2=0x7fae0c151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c070130 0x7fae0c151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 -- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c15a8b0 msgr2=0x7fae0c147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c15a8b0 0x7fae0c147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 -- 172.21.15.12:0/4129785623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c153070 con 0x7fae0c0cd950 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae09d9b640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadfc02b930 tx=0x7fadfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc03aa60 con 0x7fae0c0cd950 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc03ac00 con 0x7fae0c0cd950 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c153350 con 0x7fae0c0cd950 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0443f0 con 0x7fae0c0cd950 2024-03-08T08:28:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.316+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c153910 con 0x7fae0c0cd950 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.317+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c0749c0 con 0x7fae0c0cd950 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.318+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fadfc04b060 con 0x7fae0c0cd950 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.318+0000 7fadfaffd640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 0x7fadec06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.318+0000 7fae0a59c640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 0x7fadec06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.318+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fadfc0ab880 con 0x7fae0c0cd950 2024-03-08T08:28:59.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.319+0000 7fae0a59c640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 0x7fadec06d900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fae00009560 tx=0x7fae00009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.323+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fadfc03f070 con 0x7fae0c0cd950 2024-03-08T08:28:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.503+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae0c072cc0 con 0x7fae0c0cd950 2024-03-08T08:28:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.504+0000 7fadfaffd640 1 -- 172.21.15.12:0/4129785623 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12865 (secure 0 0 0) 0x7fadfc07a580 con 0x7fae0c0cd950 2024-03-08T08:28:59.505 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:28:59.505 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:28:59.148406+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 msgr2=0x7fadec06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 0x7fadec06d900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fae00009560 tx=0x7fae00009190 comp rx=0 tx=0).stop 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 msgr2=0x7fae0c152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadfc02b930 tx=0x7fadfc0046c0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 shutdown_connections 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fadec06b440 0x7fadec06d900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae0c15a8b0 0x7fae0c147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0cd950 0x7fae0c152900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 --2- 172.21.15.12:0/4129785623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae0c070130 0x7fae0c151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 >> 172.21.15.12:0/4129785623 conn(0x7fae0c06f0d0 msgr2=0x7fae0c0afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 shutdown_connections 2024-03-08T08:28:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.506+0000 7fae10cff640 1 -- 172.21.15.12:0/4129785623 wait complete. 2024-03-08T08:28:59.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:28:59.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:28:59.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.620+0000 7fb727ad4640 1 Processor -- start 2024-03-08T08:28:59.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.620+0000 7fb727ad4640 1 -- start start 2024-03-08T08:28:59.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb727ad4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb72015e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb727ad4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72015e7b0 con 0x7fb72006cce0 2024-03-08T08:28:59.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb727ad4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72015ea90 con 0x7fb72009a570 2024-03-08T08:28:59.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb727ad4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72015ed70 con 0x7fb72015dd70 2024-03-08T08:28:59.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb726071640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb72006cce0 0x7fb7200cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44926/0 (socket says 172.21.15.12:44926) 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.621+0000 7fb726071640 1 -- 172.21.15.12:0/346242180 learned_addr learned my addr 172.21.15.12:0/346242180 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb725870640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb72015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631723311 0 0) 0x7fb72015e7b0 con 0x7fb72006cce0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708003520 con 0x7fb72006cce0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2793767699 0 0) 0x7fb708003520 con 0x7fb72006cce0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 msgr2=0x7fb72015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb72015e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 >> v1:172.21.15.156:6789/0 conn(0x7fb72009a570 legacy=0x7fb72006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72015f050 con 0x7fb72006cce0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fb7100019f0 con 0x7fb72006cce0 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 >> v1:172.21.15.12:6789/0 conn(0x7fb72006cce0 legacy=0x7fb7200cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7080039e0 0x7fb708003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb708004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb708004d40 0x7fb7080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72015ed70 con 0x7fb708004d40 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70800baf0 con 0x7fb7080039e0 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70800bcf0 con 0x7fb72015dd70 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb708004d40 0x7fb7080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb725870640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7080039e0 0x7fb708003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb72506f640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb708004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 -- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 msgr2=0x7fb708004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb72015dd70 0x7fb708004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 -- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7080039e0 msgr2=0x7fb708003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7080039e0 0x7fb708003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb726071640 1 -- 172.21.15.12:0/346242180 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708003520 con 0x7fb708004d40 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.622+0000 7fb725870640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7080039e0 0x7fb708003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.623+0000 7fb726071640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb708004d40 0x7fb7080051c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb71000a860 tx=0x7fb708005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8570ffc1e58c46da server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.623+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71000a480 con 0x7fb708004d40 2024-03-08T08:28:59.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.623+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710034630 con 0x7fb708004d40 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.623+0000 7fb72486e640 1 -- 172.21.15.12:0/346242180 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710034a60 con 0x7fb708004d40 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 -- 172.21.15.12:0/346242180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb708004d40 msgr2=0x7fb7080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 --2- 172.21.15.12:0/346242180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb708004d40 0x7fb7080051c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb71000a860 tx=0x7fb708005830 comp rx=0 tx=0).stop 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb726071640 1 -- 172.21.15.12:0/346242180 reap_dead start 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 -- 172.21.15.12:0/346242180 shutdown_connections 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 -- 172.21.15.12:0/346242180 >> 172.21.15.12:0/346242180 conn(0x7fb7200b3370 msgr2=0x7fb7200b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 -- 172.21.15.12:0/346242180 shutdown_connections 2024-03-08T08:28:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.624+0000 7fb727ad4640 1 -- 172.21.15.12:0/346242180 wait complete. 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.625+0000 7fb727ad4640 1 Processor -- start 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.625+0000 7fb727ad4640 1 -- start start 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb720158e00 0x7fb72015be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb72015a7d0 0x7fb72015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72009ab50 con 0x7fb7200b68a0 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72009a9d0 con 0x7fb72015a7d0 2024-03-08T08:28:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72015e150 con 0x7fb720158e00 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53806/0 (socket says 172.21.15.12:53806) 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 -- 172.21.15.12:0/2573510219 learned_addr learned my addr 172.21.15.12:0/2573510219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb72506f640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb72015a7d0 0x7fb72015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb725870640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb720158e00 0x7fb72015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 -- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb720158e00 msgr2=0x7fb72015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb720158e00 0x7fb72015be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 -- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb72015a7d0 msgr2=0x7fb72015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb72015a7d0 0x7fb72015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 -- 172.21.15.12:0/2573510219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720159a10 con 0x7fb7200b68a0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb726071640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb71000a980 tx=0x7fb72015ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718009690 con 0x7fb7200b68a0 2024-03-08T08:28:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710045070 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.626+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb720159cf0 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.627+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71000a480 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.627+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7200c97a0 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.628+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7200749c0 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.628+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fb7100349e0 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.628+0000 7fb71e7fc640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 0x7fb70c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.628+0000 7fb725870640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 0x7fb70c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.628+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fb7100abdc0 con 0x7fb7200b68a0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.629+0000 7fb725870640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 0x7fb70c06d900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7200b4530 tx=0x7fb718008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.633+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb71007aff0 con 0x7fb7200b68a0 2024-03-08T08:28:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.815+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7200b4210 con 0x7fb7200b68a0 2024-03-08T08:28:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.815+0000 7fb71e7fc640 1 -- 172.21.15.12:0/2573510219 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb71007a8f0 con 0x7fb7200b68a0 2024-03-08T08:28:59.816 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:28:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 msgr2=0x7fb70c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 0x7fb70c06d900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7200b4530 tx=0x7fb718008040 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 msgr2=0x7fb7200b6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb71000a980 tx=0x7fb72015ec60 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 shutdown_connections 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb70c06b440 0x7fb70c06d900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7200b68a0 0x7fb7200b6d20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb72015a7d0 0x7fb72015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 --2- 172.21.15.12:0/2573510219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb720158e00 0x7fb72015be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 >> 172.21.15.12:0/2573510219 conn(0x7fb7200b3370 msgr2=0x7fb7200cdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 shutdown_connections 2024-03-08T08:28:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.817+0000 7fb727ad4640 1 -- 172.21.15.12:0/2573510219 wait complete. 2024-03-08T08:28:59.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:28:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.929+0000 7fa990ad4640 1 Processor -- start 2024-03-08T08:28:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.929+0000 7fa990ad4640 1 -- start start 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa990ad4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 0x7fa98c0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa990ad4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa98c0d31c0 0x7fa98c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa990ad4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c15e1a0 con 0x7fa98c15ddc0 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa990ad4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0b9030 con 0x7fa98c0d31c0 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa990ad4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0d1d20 con 0x7fa98c0d1110 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa98c0d31c0 0x7fa98c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa98c0d31c0 0x7fa98c0b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53816/0 (socket says 172.21.15.12:53816) 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 -- 172.21.15.12:0/3983976680 learned_addr learned my addr 172.21.15.12:0/3983976680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa983fff640 1 --2- 172.21.15.12:0/3983976680 >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 0x7fa98c0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 -- 172.21.15.12:0/3983976680 >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 msgr2=0x7fa98c0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 --2- 172.21.15.12:0/3983976680 >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 0x7fa98c0b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 -- 172.21.15.12:0/3983976680 >> v1:172.21.15.156:6789/0 conn(0x7fa98c15ddc0 legacy=0x7fa98c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa98ad9d640 1 -- 172.21.15.12:0/3983976680 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0b8790 con 0x7fa98c0d31c0 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.930+0000 7fa983fff640 1 --2- 172.21.15.12:0/3983976680 >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 0x7fa98c0b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa98ad9d640 1 --2- 172.21.15.12:0/3983976680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c0b7120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0b6420 tx=0x7fa98402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619d3b27acbd102 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa989d9b640 1 -- 172.21.15.12:0/3983976680 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98402ca80 con 0x7fa98c0d31c0 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa989d9b640 1 -- 172.21.15.12:0/3983976680 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98402cc20 con 0x7fa98c0d31c0 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa989d9b640 1 -- 172.21.15.12:0/3983976680 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9840343f0 con 0x7fa98c0d31c0 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 -- 172.21.15.12:0/3983976680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 msgr2=0x7fa98c0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3983976680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c0b7120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0b6420 tx=0x7fa98402bee0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 -- 172.21.15.12:0/3983976680 shutdown_connections 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3983976680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c0b7120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3983976680 >> v2:172.21.15.156:3300/0 conn(0x7fa98c0d1110 0x7fa98c0b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 -- 172.21.15.12:0/3983976680 >> 172.21.15.12:0/3983976680 conn(0x7fa98c072720 msgr2=0x7fa98c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:28:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 -- 172.21.15.12:0/3983976680 shutdown_connections 2024-03-08T08:28:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.931+0000 7fa990ad4640 1 -- 172.21.15.12:0/3983976680 wait complete. 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 Processor -- start 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 -- start start 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c0d1110 0x7fa98c0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa98c15ddc0 0x7fa98c1e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b9980 con 0x7fa98c0d31c0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b8d50 con 0x7fa98c0d1110 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.933+0000 7fa990ad4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b7660 con 0x7fa98c15ddc0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53824/0 (socket says 172.21.15.12:53824) 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 -- 172.21.15.12:0/3430047318 learned_addr learned my addr 172.21.15.12:0/3430047318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa98ad9d640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa98c15ddc0 0x7fa98c1e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa98a59c640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c0d1110 0x7fa98c0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 -- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa98c15ddc0 msgr2=0x7fa98c1e8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa98c15ddc0 0x7fa98c1e8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 -- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c0d1110 msgr2=0x7fa98c0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c0d1110 0x7fa98c0c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 -- 172.21.15.12:0/3430047318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c160ce0 con 0x7fa98c0d31c0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa983fff640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c7f0 tx=0x7fa97400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974015070 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa974007a30 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c160720 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974007e00 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.934+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c148aa0 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.935+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c0749c0 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.935+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fa974018070 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.936+0000 7fa982ffd640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 0x7fa96c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.936+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fa974089c40 con 0x7fa98c0d31c0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.936+0000 7fa98a59c640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 0x7fa96c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:28:59.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.936+0000 7fa98a59c640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 0x7fa96c06d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa98c161370 tx=0x7fa97c009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:28:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:28:59.940+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa974090080 con 0x7fa98c0d31c0 2024-03-08T08:29:00.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.161+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa98c0b5d60 con 0x7fa98c0d31c0 2024-03-08T08:29:00.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.162+0000 7fa982ffd640 1 -- 172.21.15.12:0/3430047318 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fa9740588b0 con 0x7fa98c0d31c0 2024-03-08T08:29:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:00.163 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-08T08:29:00.163 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-03-08T08:29:00.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 msgr2=0x7fa96c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:00.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 0x7fa96c06d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa98c161370 tx=0x7fa97c009060 comp rx=0 tx=0).stop 2024-03-08T08:29:00.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 msgr2=0x7fa98c15fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:00.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c7f0 tx=0x7fa97400cbf0 comp rx=0 tx=0).stop 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 shutdown_connections 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa96c06b440 0x7fa96c06d900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa98c15ddc0 0x7fa98c1e8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0d31c0 0x7fa98c15fdd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 --2- 172.21.15.12:0/3430047318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c0d1110 0x7fa98c0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 >> 172.21.15.12:0/3430047318 conn(0x7fa98c072720 msgr2=0x7fa98c0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 shutdown_connections 2024-03-08T08:29:00.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:00.164+0000 7fa990ad4640 1 -- 172.21.15.12:0/3430047318 wait complete. 2024-03-08T08:29:02.178 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.290+0000 7f0ee8e83640 1 Processor -- start 2024-03-08T08:29:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.290+0000 7f0ee8e83640 1 -- start start 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee8e83640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee406c240 0x7f0ee406c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee8e83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee40d24b0 0x7f0ee40d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee8e83640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee415e930 con 0x7f0ee40b9420 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee8e83640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee40b6b30 con 0x7f0ee406c240 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee8e83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee406cb60 con 0x7f0ee40d24b0 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee406c240 0x7f0ee406c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee406c240 0x7f0ee406c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53862/0 (socket says 172.21.15.12:53862) 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 -- 172.21.15.12:0/1267106619 learned_addr learned my addr 172.21.15.12:0/1267106619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee2d9d640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee40d24b0 0x7f0ee40d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 -- 172.21.15.12:0/1267106619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee40d24b0 msgr2=0x7f0ee40d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee40d24b0 0x7f0ee40d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 -- 172.21.15.12:0/1267106619 >> v1:172.21.15.156:6789/0 conn(0x7f0ee40b9420 legacy=0x7f0ee415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee159a640 1 -- 172.21.15.12:0/1267106619 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247282648 0 0) 0x7f0ee415e930 con 0x7f0ee40b9420 2024-03-08T08:29:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.291+0000 7f0ee259c640 1 -- 172.21.15.12:0/1267106619 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee415ebc0 con 0x7f0ee406c240 2024-03-08T08:29:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.292+0000 7f0ee259c640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee406c240 0x7f0ee406c620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009870 tx=0x7f0ed402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecac0581f08ee27b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.292+0000 7f0ee159a640 1 -- 172.21.15.12:0/1267106619 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed402f840 con 0x7f0ee406c240 2024-03-08T08:29:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.292+0000 7f0ee159a640 1 -- 172.21.15.12:0/1267106619 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed402f9e0 con 0x7f0ee406c240 2024-03-08T08:29:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.292+0000 7f0ee159a640 1 -- 172.21.15.12:0/1267106619 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed402fdb0 con 0x7f0ee406c240 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 -- 172.21.15.12:0/1267106619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee406c240 msgr2=0x7f0ee406c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee406c240 0x7f0ee406c620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009870 tx=0x7f0ed402ba20 comp rx=0 tx=0).stop 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 -- 172.21.15.12:0/1267106619 shutdown_connections 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee40d24b0 0x7f0ee40d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/1267106619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee406c240 0x7f0ee406c620 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 -- 172.21.15.12:0/1267106619 >> 172.21.15.12:0/1267106619 conn(0x7f0ee4072900 msgr2=0x7f0ee40d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 -- 172.21.15.12:0/1267106619 shutdown_connections 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.293+0000 7f0ee8e83640 1 -- 172.21.15.12:0/1267106619 wait complete. 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.294+0000 7f0ee8e83640 1 Processor -- start 2024-03-08T08:29:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.294+0000 7f0ee8e83640 1 -- start start 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee406c240 0x7f0ee4154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee40b9420 0x7f0ee40b8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40d1f90 con 0x7f0ee40d24b0 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40d1cb0 con 0x7f0ee40b9420 2024-03-08T08:29:02.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee8e83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee415bc30 con 0x7f0ee406c240 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53868/0 (socket says 172.21.15.12:53868) 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 -- 172.21.15.12:0/163636794 learned_addr learned my addr 172.21.15.12:0/163636794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee259c640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee406c240 0x7f0ee4154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee1d9b640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee40b9420 0x7f0ee40b8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 -- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee406c240 msgr2=0x7f0ee4154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee406c240 0x7f0ee4154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 -- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee40b9420 msgr2=0x7f0ee40b8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee40b9420 0x7f0ee40b8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.295+0000 7f0ee2d9d640 1 -- 172.21.15.12:0/163636794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4154740 con 0x7f0ee40d24b0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ee2d9d640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009dd0 tx=0x7f0edc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc018020 con 0x7f0ee40d24b0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc014480 con 0x7f0ee40d24b0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee40b36d0 con 0x7f0ee40d24b0 2024-03-08T08:29:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc014870 con 0x7f0ee40d24b0 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.296+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee40b3c90 con 0x7f0ee40d24b0 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.297+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee40749c0 con 0x7f0ee40d24b0 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.297+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0edc005d10 con 0x7f0ee40d24b0 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.298+0000 7f0ed2ffd640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.298+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0edc08aa60 con 0x7f0ee40d24b0 2024-03-08T08:29:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.298+0000 7f0ee259c640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.298+0000 7f0ee259c640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4036000 tx=0x7f0ed4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.302+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0edc022070 con 0x7f0ee40d24b0 2024-03-08T08:29:02.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.519+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ee40b3210 con 0x7f0ee40d24b0 2024-03-08T08:29:02.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.519+0000 7f0ed2ffd640 1 -- 172.21.15.12:0/163636794 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f0edc010040 con 0x7f0ee40d24b0 2024-03-08T08:29:02.520 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:02.521 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-08T08:29:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 msgr2=0x7f0ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4036000 tx=0x7f0ed4008030 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 msgr2=0x7f0ee40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009dd0 tx=0x7f0edc00c490 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 shutdown_connections 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee40d24b0 0x7f0ee40b8b90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee40b9420 0x7f0ee40b8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 --2- 172.21.15.12:0/163636794 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ee406c240 0x7f0ee4154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 >> 172.21.15.12:0/163636794 conn(0x7f0ee4072900 msgr2=0x7f0ee415b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 shutdown_connections 2024-03-08T08:29:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.521+0000 7f0ee8e83640 1 -- 172.21.15.12:0/163636794 wait complete. 2024-03-08T08:29:02.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.633+0000 7f95c6b5c640 1 Processor -- start 2024-03-08T08:29:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.633+0000 7f95c6b5c640 1 -- start start 2024-03-08T08:29:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c6b5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c00d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c6b5c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d0f40 0x7f95c00d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c6b5c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c015e1a0 con 0x7f95c00d26f0 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c6b5c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c00b9030 con 0x7f95c00d0f40 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c6b5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00d13a0 con 0x7f95c015ddc0 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d0f40 0x7f95c00d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d0f40 0x7f95c00d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53882/0 (socket says 172.21.15.12:53882) 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c50f9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f95c00d26f0 0x7f95c00d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44942/0 (socket says 172.21.15.12:44942) 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 -- 172.21.15.12:0/2278670625 learned_addr learned my addr 172.21.15.12:0/2278670625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c48f8640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c00d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 -- 172.21.15.12:0/2278670625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 msgr2=0x7f95c00d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c00d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 -- 172.21.15.12:0/2278670625 >> v1:172.21.15.12:6789/0 conn(0x7f95c00d26f0 legacy=0x7f95c00d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95bffff640 1 -- 172.21.15.12:0/2278670625 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00b8790 con 0x7f95c00d0f40 2024-03-08T08:29:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.634+0000 7f95c48f8640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c00d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.635+0000 7f95bffff640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00d21b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009870 tx=0x7f95ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffb486594afe46a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.635+0000 7f95bf7fe640 1 -- 172.21.15.12:0/2278670625 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac034070 con 0x7f95c00d0f40 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.635+0000 7f95bf7fe640 1 -- 172.21.15.12:0/2278670625 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac030840 con 0x7f95c00d0f40 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.635+0000 7f95bf7fe640 1 -- 172.21.15.12:0/2278670625 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac030c10 con 0x7f95c00d0f40 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 -- 172.21.15.12:0/2278670625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 msgr2=0x7f95c00d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00d21b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009870 tx=0x7f95ac02b980 comp rx=0 tx=0).stop 2024-03-08T08:29:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 -- 172.21.15.12:0/2278670625 shutdown_connections 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00d21b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/2278670625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c00d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 -- 172.21.15.12:0/2278670625 >> 172.21.15.12:0/2278670625 conn(0x7f95c0072720 msgr2=0x7f95c0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 -- 172.21.15.12:0/2278670625 shutdown_connections 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.636+0000 7f95c6b5c640 1 -- 172.21.15.12:0/2278670625 wait complete. 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.637+0000 7f95c6b5c640 1 Processor -- start 2024-03-08T08:29:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.637+0000 7f95c6b5c640 1 -- start start 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c00d26f0 0x7f95c00cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c0157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b8d50 con 0x7f95c00d0f40 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00d1d20 con 0x7f95c00d26f0 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c6b5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b9980 con 0x7f95c015ddc0 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53890/0 (socket says 172.21.15.12:53890) 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 -- 172.21.15.12:0/3200277806 learned_addr learned my addr 172.21.15.12:0/3200277806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95bffff640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c00d26f0 0x7f95c00cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c50f9640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c0157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 -- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 msgr2=0x7f95c0157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c0157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 -- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c00d26f0 msgr2=0x7f95c00cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c00d26f0 0x7f95c00cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 -- 172.21.15.12:0/3200277806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00cb2f0 con 0x7f95c00d0f40 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.638+0000 7f95c48f8640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c7f0 tx=0x7f95b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.639+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4015070 con 0x7f95c00d0f40 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.639+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b4007820 con 0x7f95c00d0f40 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.639+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c00cb5d0 con 0x7f95c00d0f40 2024-03-08T08:29:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.639+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4007bd0 con 0x7f95c00d0f40 2024-03-08T08:29:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.639+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c00cbb90 con 0x7f95c00d0f40 2024-03-08T08:29:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.640+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c00749c0 con 0x7f95c00d0f40 2024-03-08T08:29:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.640+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95b4018070 con 0x7f95c00d0f40 2024-03-08T08:29:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.641+0000 7f95bd7fa640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 0x7f95a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.641+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f95b4089fc0 con 0x7f95c00d0f40 2024-03-08T08:29:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.641+0000 7f95bffff640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 0x7f95a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.641+0000 7f95bffff640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 0x7f95a406d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f95ac036000 tx=0x7f95ac03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.645+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f95b40591a0 con 0x7f95c00d0f40 2024-03-08T08:29:02.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.826+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95c01615e0 con 0x7f95c00d0f40 2024-03-08T08:29:02.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.827+0000 7f95bd7fa640 1 -- 172.21.15.12:0/3200277806 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f95b4058aa0 con 0x7f95c00d0f40 2024-03-08T08:29:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:02.828 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:01.212555+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.828+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 msgr2=0x7f95a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 0x7f95a406d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f95ac036000 tx=0x7f95ac03b040 comp rx=0 tx=0).stop 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 msgr2=0x7f95c00cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c7f0 tx=0x7f95b400ccb0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 shutdown_connections 2024-03-08T08:29:02.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f95a406b4d0 0x7f95a406d990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95c015ddc0 0x7f95c0157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c00d26f0 0x7f95c00cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 --2- 172.21.15.12:0/3200277806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d0f40 0x7f95c00cd5f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 >> 172.21.15.12:0/3200277806 conn(0x7f95c0072720 msgr2=0x7f95c00b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 shutdown_connections 2024-03-08T08:29:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.829+0000 7f95c6b5c640 1 -- 172.21.15.12:0/3200277806 wait complete. 2024-03-08T08:29:02.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:02.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:02.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.943+0000 7f38c0071640 1 Processor -- start 2024-03-08T08:29:02.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.943+0000 7f38c0071640 1 -- start start 2024-03-08T08:29:02.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b815ddc0 0x7f38b80d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 0x7f38b80d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38b80d26f0 0x7f38b80d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80b9030 con 0x7f38b80d26f0 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80d13a0 con 0x7f38b80d0f40 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38c0071640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80d1d20 con 0x7f38b815ddc0 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38b80d26f0 0x7f38b80d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38b80d26f0 0x7f38b80d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53904/0 (socket says 172.21.15.12:53904) 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 -- 172.21.15.12:0/2869166704 learned_addr learned my addr 172.21.15.12:0/2869166704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38bd60c640 1 --2- 172.21.15.12:0/2869166704 >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 0x7f38b80d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38bde0d640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b815ddc0 0x7f38b80d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 -- 172.21.15.12:0/2869166704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b815ddc0 msgr2=0x7f38b80d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b815ddc0 0x7f38b80d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 -- 172.21.15.12:0/2869166704 >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 msgr2=0x7f38b80d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 --2- 172.21.15.12:0/2869166704 >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 0x7f38b80d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38be60e640 1 -- 172.21.15.12:0/2869166704 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b815e1a0 con 0x7f38b80d26f0 2024-03-08T08:29:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.944+0000 7f38bd60c640 1 --2- 172.21.15.12:0/2869166704 >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 0x7f38b80d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.945+0000 7f38be60e640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80d2b80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38b808ccc0 tx=0x7f38a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c2a4d36c9d31e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.945+0000 7f38bce0b640 1 -- 172.21.15.12:0/2869166704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a802ca70 con 0x7f38b80d26f0 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.945+0000 7f38bce0b640 1 -- 172.21.15.12:0/2869166704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a802cc10 con 0x7f38b80d26f0 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.945+0000 7f38bce0b640 1 -- 172.21.15.12:0/2869166704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a80343f0 con 0x7f38b80d26f0 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 -- 172.21.15.12:0/2869166704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 msgr2=0x7f38b80d2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80d2b80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38b808ccc0 tx=0x7f38a802bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 -- 172.21.15.12:0/2869166704 shutdown_connections 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80d2b80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 --2- 172.21.15.12:0/2869166704 >> v2:172.21.15.156:3300/0 conn(0x7f38b80d0f40 0x7f38b80d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 --2- 172.21.15.12:0/2869166704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b815ddc0 0x7f38b80d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 -- 172.21.15.12:0/2869166704 >> 172.21.15.12:0/2869166704 conn(0x7f38b8072720 msgr2=0x7f38b8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 -- 172.21.15.12:0/2869166704 shutdown_connections 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.946+0000 7f38c0071640 1 -- 172.21.15.12:0/2869166704 wait complete. 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.947+0000 7f38c0071640 1 Processor -- start 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.947+0000 7f38c0071640 1 -- start start 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b80d0f40 0x7f38b815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 0x7f38b80aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80b8d50 con 0x7f38b80d26f0 2024-03-08T08:29:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80b8a70 con 0x7f38b815ddc0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80b9980 con 0x7f38b80d0f40 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53914/0 (socket says 172.21.15.12:53914) 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 -- 172.21.15.12:0/3211611161 learned_addr learned my addr 172.21.15.12:0/3211611161 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38be60e640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 0x7f38b80aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bde0d640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b80d0f40 0x7f38b815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 -- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b80d0f40 msgr2=0x7f38b815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b80d0f40 0x7f38b815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 -- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 msgr2=0x7f38b80aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 0x7f38b80aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 -- 172.21.15.12:0/3211611161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b815efd0 con 0x7f38b80d26f0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38be60e640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 0x7f38b80aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38bd60c640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e730 tx=0x7f38ac00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00f040 con 0x7f38b80d26f0 2024-03-08T08:29:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac004230 con 0x7f38b80d26f0 2024-03-08T08:29:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b815f2b0 con 0x7f38b80d26f0 2024-03-08T08:29:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00ee40 con 0x7f38b80d26f0 2024-03-08T08:29:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.948+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b815a680 con 0x7f38b80d26f0 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.949+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b80749c0 con 0x7f38b80d26f0 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.950+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38ac00c6d0 con 0x7f38b80d26f0 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.950+0000 7f38b67fc640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 0x7f38a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.950+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f38ac089b60 con 0x7f38b80d26f0 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.950+0000 7f38bde0d640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 0x7f38a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:02.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.951+0000 7f38bde0d640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 0x7f38a406d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38b0006fd0 tx=0x7f38b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:02.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:02.954+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f38ac017020 con 0x7f38b80d26f0 2024-03-08T08:29:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.133+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38b80b67c0 con 0x7f38b80d26f0 2024-03-08T08:29:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.133+0000 7f38b67fc640 1 -- 172.21.15.12:0/3211611161 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38ac058dc0 con 0x7f38b80d26f0 2024-03-08T08:29:03.134 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 msgr2=0x7f38a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 0x7f38a406d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38b0006fd0 tx=0x7f38b0008040 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 msgr2=0x7f38b80cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e730 tx=0x7f38ac00ebf0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 shutdown_connections 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f38a406b500 0x7f38a406d9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b815ddc0 0x7f38b80aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38b80d26f0 0x7f38b80cd130 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 --2- 172.21.15.12:0/3211611161 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f38b80d0f40 0x7f38b815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 >> 172.21.15.12:0/3211611161 conn(0x7f38b8072720 msgr2=0x7f38b80d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 shutdown_connections 2024-03-08T08:29:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.135+0000 7f38c0071640 1 -- 172.21.15.12:0/3211611161 wait complete. 2024-03-08T08:29:03.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-08T08:29:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.247+0000 7fb5a7fc2640 1 Processor -- start 2024-03-08T08:29:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.247+0000 7fb5a7fc2640 1 -- start start 2024-03-08T08:29:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a7fc2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a00d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a7fc2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb5a00d0f40 0x7fb5a00d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a7fc2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a015e1a0 con 0x7fb5a00d26f0 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a7fc2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00b9030 con 0x7fb5a00d0f40 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a7fc2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00d13a0 con 0x7fb5a015ddc0 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a555d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb5a00d0f40 0x7fb5a00d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a655f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb5a00d26f0 0x7fb5a00d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44956/0 (socket says 172.21.15.12:44956) 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a655f640 1 -- 172.21.15.12:0/1000113485 learned_addr learned my addr 172.21.15.12:0/1000113485 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a5d5e640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a00d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a555d640 1 -- 172.21.15.12:0/1000113485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 msgr2=0x7fb5a00d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a555d640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a00d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a555d640 1 -- 172.21.15.12:0/1000113485 >> v1:172.21.15.12:6789/0 conn(0x7fb5a00d26f0 legacy=0x7fb5a00d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a555d640 1 -- 172.21.15.12:0/1000113485 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00b8790 con 0x7fb5a00d0f40 2024-03-08T08:29:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a4d5c640 1 -- 172.21.15.12:0/1000113485 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057063084 0 0) 0x7fb5a015e1a0 con 0x7fb5a00d26f0 2024-03-08T08:29:03.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.248+0000 7fb5a5d5e640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a00d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:03.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a555d640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a00d21b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097a0 tx=0x7fb59002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0a827f6fd24e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a4d5c640 1 -- 172.21.15.12:0/1000113485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590035020 con 0x7fb5a00d0f40 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a4d5c640 1 -- 172.21.15.12:0/1000113485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb590031830 con 0x7fb5a00d0f40 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a4d5c640 1 -- 172.21.15.12:0/1000113485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590031c00 con 0x7fb5a00d0f40 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/1000113485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 msgr2=0x7fb5a00d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a00d21b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097a0 tx=0x7fb59002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/1000113485 shutdown_connections 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a00d21b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/1000113485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a00d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.249+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/1000113485 >> 172.21.15.12:0/1000113485 conn(0x7fb5a0072720 msgr2=0x7fb5a0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.250+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/1000113485 shutdown_connections 2024-03-08T08:29:03.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.250+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/1000113485 wait complete. 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 Processor -- start 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 -- start start 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5a00d26f0 0x7fb5a00ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00b8d50 con 0x7fb5a00d0f40 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00d1d20 con 0x7fb5a00d26f0 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a7fc2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00b9980 con 0x7fb5a015ddc0 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a5d5e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a5d5e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53944/0 (socket says 172.21.15.12:53944) 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.251+0000 7fb5a5d5e640 1 -- 172.21.15.12:0/664468398 learned_addr learned my addr 172.21.15.12:0/664468398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 -- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 msgr2=0x7fb5a015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a655f640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a555d640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5a00d26f0 0x7fb5a00ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a015f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 -- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5a00d26f0 msgr2=0x7fb5a00ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5a00d26f0 0x7fb5a00ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 -- 172.21.15.12:0/664468398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00aa5e0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a5d5e640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c820 tx=0x7fb59c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c015070 con 0x7fb5a00d0f40 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c007a30 con 0x7fb5a00d0f40 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a00aa8c0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c007de0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.252+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a00aae80 con 0x7fb5a00d0f40 2024-03-08T08:29:03.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.253+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a00749c0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.253+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb59c018070 con 0x7fb5a00d0f40 2024-03-08T08:29:03.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.254+0000 7fb5967fc640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.254+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb59c089d50 con 0x7fb5a00d0f40 2024-03-08T08:29:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.254+0000 7fb5a555d640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.254+0000 7fb5a555d640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 0x7fb58406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb590036000 tx=0x7fb590008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.258+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb59c08d030 con 0x7fb5a00d0f40 2024-03-08T08:29:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.440+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb5a00d24d0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.440+0000 7fb5967fc640 1 -- 172.21.15.12:0/664468398 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb59c058fb0 con 0x7fb5a00d0f40 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:29:03.442 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 msgr2=0x7fb58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 0x7fb58406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb590036000 tx=0x7fb590008030 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 msgr2=0x7fb5a015ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c820 tx=0x7fb59c00cce0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 shutdown_connections 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb5a015ddc0 0x7fb5a015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5a00d26f0 0x7fb5a00ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 --2- 172.21.15.12:0/664468398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a00d0f40 0x7fb5a015ae90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 >> 172.21.15.12:0/664468398 conn(0x7fb5a0072720 msgr2=0x7fb5a015d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 shutdown_connections 2024-03-08T08:29:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.442+0000 7fb5a7fc2640 1 -- 172.21.15.12:0/664468398 wait complete. 2024-03-08T08:29:03.454 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-08T08:29:03.454 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-08T08:29:03.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.557+0000 7f1fe6215640 1 Processor -- start 2024-03-08T08:29:03.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.557+0000 7f1fe6215640 1 -- start start 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fe6215640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fe015bd60 0x7f1fe0099b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fe6215640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009a4f0 con 0x7f1fe009a0d0 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fe6215640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00cd3d0 con 0x7f1fe00ccf10 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fe6215640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00cd6b0 con 0x7f1fe015bd60 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fdf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fe015bd60 0x7f1fe0099b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fdf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fe015bd60 0x7f1fe0099b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53950/0 (socket says 172.21.15.12:53950) 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.558+0000 7f1fdf7fe640 1 -- 172.21.15.12:0/1709508974 learned_addr learned my addr 172.21.15.12:0/1709508974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.559+0000 7f1fdf7fe640 1 -- 172.21.15.12:0/1709508974 >> v1:172.21.15.156:6789/0 conn(0x7f1fe00ccf10 legacy=0x7f1fe00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:03.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.559+0000 7f1fdf7fe640 1 -- 172.21.15.12:0/1709508974 >> v1:172.21.15.12:6789/0 conn(0x7f1fe009a0d0 legacy=0x7f1fe00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.559+0000 7f1fdf7fe640 1 -- 172.21.15.12:0/1709508974 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00cd990 con 0x7f1fe015bd60 2024-03-08T08:29:03.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.559+0000 7f1fdf7fe640 1 --2- 172.21.15.12:0/1709508974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe015bd60 0x7f1fe0099b90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27697d775fb2d51f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.559+0000 7f1fde7fc640 1 -- 172.21.15.12:0/1709508974 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0034070 con 0x7f1fe015bd60 2024-03-08T08:29:03.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fde7fc640 1 -- 172.21.15.12:0/1709508974 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0030860 con 0x7f1fe015bd60 2024-03-08T08:29:03.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fde7fc640 1 -- 172.21.15.12:0/1709508974 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0030c30 con 0x7f1fe015bd60 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 -- 172.21.15.12:0/1709508974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe015bd60 msgr2=0x7f1fe0099b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1709508974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe015bd60 0x7f1fe0099b90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 -- 172.21.15.12:0/1709508974 shutdown_connections 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1709508974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe015bd60 0x7f1fe0099b90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 -- 172.21.15.12:0/1709508974 >> 172.21.15.12:0/1709508974 conn(0x7f1fe00b3380 msgr2=0x7f1fe00b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 -- 172.21.15.12:0/1709508974 shutdown_connections 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.560+0000 7f1fe6215640 1 -- 172.21.15.12:0/1709508974 wait complete. 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.561+0000 7f1fe6215640 1 Processor -- start 2024-03-08T08:29:03.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.561+0000 7f1fe6215640 1 -- start start 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 0x7f1fe00c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 0x7f1fe01f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00d0470 con 0x7f1fe00ccf10 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b0e30 con 0x7f1fe015bd60 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fe6215640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00d05f0 con 0x7f1fe009a0d0 2024-03-08T08:29:03.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53962/0 (socket says 172.21.15.12:53962) 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 -- 172.21.15.12:0/1633141667 learned_addr learned my addr 172.21.15.12:0/1633141667 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdffff640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 0x7f1fe01f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdf7fe640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 0x7f1fe00c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 -- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 msgr2=0x7f1fe00c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 0x7f1fe00c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 -- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 msgr2=0x7f1fe01f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 0x7f1fe01f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.562+0000 7f1fdeffd640 1 -- 172.21.15.12:0/1633141667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe015e4a0 con 0x7f1fe00ccf10 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fdffff640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 0x7f1fe01f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fdf7fe640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 0x7f1fe00c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:03.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fdeffd640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400c400 tx=0x7f1fd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4011790 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd4017070 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe015e780 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4011bb0 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.563+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe015ed40 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.564+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe00749c0 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.564+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd401c460 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.565+0000 7f1fc7fff640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 0x7f1fcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.565+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1fd4089c70 con 0x7f1fe00ccf10 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.565+0000 7f1fdf7fe640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 0x7f1fcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:03.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.565+0000 7f1fdf7fe640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 0x7f1fcc06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0036000 tx=0x7f1fd003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:03.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.569+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1fd4058e40 con 0x7f1fe00ccf10 2024-03-08T08:29:03.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:03.748+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1fe00712c0 con 0x7f1fe00ccf10 2024-03-08T08:29:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.223+0000 7f1fc7fff640 1 -- 172.21.15.12:0/1633141667 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1fd4058740 con 0x7f1fe00ccf10 2024-03-08T08:29:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 msgr2=0x7f1fcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 0x7f1fcc06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0036000 tx=0x7f1fd003b040 comp rx=0 tx=0).stop 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 msgr2=0x7f1fe015dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400c400 tx=0x7f1fd400c8c0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 shutdown_connections 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fcc06b4d0 0x7f1fcc06d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fe015bd60 0x7f1fe01f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe00ccf10 0x7f1fe015dd50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 --2- 172.21.15.12:0/1633141667 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fe009a0d0 0x7f1fe00c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.227+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 >> 172.21.15.12:0/1633141667 conn(0x7f1fe00b3380 msgr2=0x7f1fe00cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.228+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 shutdown_connections 2024-03-08T08:29:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.228+0000 7f1fe6215640 1 -- 172.21.15.12:0/1633141667 wait complete. 2024-03-08T08:29:04.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-08T08:29:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.354+0000 7fef5702e640 1 Processor -- start 2024-03-08T08:29:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.354+0000 7fef5702e640 1 -- start start 2024-03-08T08:29:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef5702e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef500b9760 0x7fef500d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef5702e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 0x7fef500b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef5702e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef500b7250 con 0x7fef5015dbc0 2024-03-08T08:29:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef5702e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef500b7520 con 0x7fef500b9760 2024-03-08T08:29:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef5702e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef500b7800 con 0x7fef500d2050 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef500b9760 0x7fef500d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef54dca640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fef5015dbc0 0x7fef500b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44970/0 (socket says 172.21.15.12:44970) 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef54dca640 1 -- 172.21.15.12:0/1500857488 learned_addr learned my addr 172.21.15.12:0/1500857488 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef555cb640 1 --2- 172.21.15.12:0/1500857488 >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 0x7fef500b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4ffff640 1 -- 172.21.15.12:0/1500857488 >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 msgr2=0x7fef500b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4ffff640 1 --2- 172.21.15.12:0/1500857488 >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 0x7fef500b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4ffff640 1 -- 172.21.15.12:0/1500857488 >> v1:172.21.15.12:6789/0 conn(0x7fef5015dbc0 legacy=0x7fef500b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4ffff640 1 -- 172.21.15.12:0/1500857488 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef500b7ae0 con 0x7fef500b9760 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef4f7fe640 1 -- 172.21.15.12:0/1500857488 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938203893 0 0) 0x7fef500b7250 con 0x7fef5015dbc0 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.355+0000 7fef555cb640 1 --2- 172.21.15.12:0/1500857488 >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 0x7fef500b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.356+0000 7fef4ffff640 1 --2- 172.21.15.12:0/1500857488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef500d1b10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef500ce0b0 tx=0x7fef4002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3c83b456e6e10c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.356+0000 7fef4f7fe640 1 -- 172.21.15.12:0/1500857488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40034070 con 0x7fef500b9760 2024-03-08T08:29:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.356+0000 7fef4f7fe640 1 -- 172.21.15.12:0/1500857488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef40030610 con 0x7fef500b9760 2024-03-08T08:29:04.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.356+0000 7fef4f7fe640 1 -- 172.21.15.12:0/1500857488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef400309e0 con 0x7fef500b9760 2024-03-08T08:29:04.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 -- 172.21.15.12:0/1500857488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 msgr2=0x7fef500d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 --2- 172.21.15.12:0/1500857488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef500d1b10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef500ce0b0 tx=0x7fef4002b960 comp rx=0 tx=0).stop 2024-03-08T08:29:04.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 -- 172.21.15.12:0/1500857488 shutdown_connections 2024-03-08T08:29:04.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 --2- 172.21.15.12:0/1500857488 >> v2:172.21.15.156:3300/0 conn(0x7fef500d2050 0x7fef500b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 --2- 172.21.15.12:0/1500857488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef500d1b10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 -- 172.21.15.12:0/1500857488 >> 172.21.15.12:0/1500857488 conn(0x7fef50072720 msgr2=0x7fef50072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 -- 172.21.15.12:0/1500857488 shutdown_connections 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.357+0000 7fef5702e640 1 -- 172.21.15.12:0/1500857488 wait complete. 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.358+0000 7fef5702e640 1 Processor -- start 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.358+0000 7fef5702e640 1 -- start start 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fef500d2050 0x7fef50160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef5015dbc0 0x7fef500d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5015dfa0 con 0x7fef500b9760 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef500b80a0 con 0x7fef5015dbc0 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef5702e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef500b8b00 con 0x7fef500d2050 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53982/0 (socket says 172.21.15.12:53982) 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 -- 172.21.15.12:0/1025229472 learned_addr learned my addr 172.21.15.12:0/1025229472 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef555cb640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef5015dbc0 0x7fef500d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef4ffff640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fef500d2050 0x7fef50160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 -- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fef500d2050 msgr2=0x7fef50160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fef500d2050 0x7fef50160040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 -- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef5015dbc0 msgr2=0x7fef500d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef5015dbc0 0x7fef500d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.359+0000 7fef54dca640 1 -- 172.21.15.12:0/1025229472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5015e2c0 con 0x7fef500b9760 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef54dca640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c7c0 tx=0x7fef4800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48018070 con 0x7fef500b9760 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4800a640 con 0x7fef500b9760 2024-03-08T08:29:04.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5015e5a0 con 0x7fef500b9760 2024-03-08T08:29:04.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4800aa30 con 0x7fef500b9760 2024-03-08T08:29:04.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.360+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5015eb60 con 0x7fef500b9760 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.361+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef500749c0 con 0x7fef500b9760 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.361+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef4801b070 con 0x7fef500b9760 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.362+0000 7fef4d7fa640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 0x7fef3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.362+0000 7fef4ffff640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 0x7fef3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.362+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fef480895e0 con 0x7fef500b9760 2024-03-08T08:29:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.362+0000 7fef4ffff640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 0x7fef3006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef40036000 tx=0x7fef4003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.366+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fef480586d0 con 0x7fef500b9760 2024-03-08T08:29:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:04.548+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fef500b5160 con 0x7fef500b9760 2024-03-08T08:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.228+0000 7fef4d7fa640 1 -- 172.21.15.12:0/1025229472 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fef48057fd0 con 0x7fef500b9760 2024-03-08T08:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-08T08:29:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 msgr2=0x7fef3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 0x7fef3006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef40036000 tx=0x7fef4003b040 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 msgr2=0x7fef5015a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c7c0 tx=0x7fef4800cc80 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 shutdown_connections 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fef3006b4d0 0x7fef3006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef5015dbc0 0x7fef500d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fef500d2050 0x7fef50160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 --2- 172.21.15.12:0/1025229472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef500b9760 0x7fef5015a520 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 >> 172.21.15.12:0/1025229472 conn(0x7fef50072720 msgr2=0x7fef500b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 shutdown_connections 2024-03-08T08:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.231+0000 7fef5702e640 1 -- 172.21.15.12:0/1025229472 wait complete. 2024-03-08T08:29:05.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-08T08:29:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.365+0000 7f046fd48640 1 Processor -- start 2024-03-08T08:29:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.365+0000 7f046fd48640 1 -- start start 2024-03-08T08:29:05.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046fd48640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f04680b8300 0x7f04681688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046fd48640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046815e1a0 con 0x7f046815ddc0 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046fd48640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b8790 con 0x7f04680d1110 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046fd48640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04680b9030 con 0x7f04680b8300 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046dae4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f046815ddc0 0x7f04680d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44974/0 (socket says 172.21.15.12:44974) 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046dae4640 1 -- 172.21.15.12:0/133123074 learned_addr learned my addr 172.21.15.12:0/133123074 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266262233 0 0) 0x7f046815e1a0 con 0x7f046815ddc0 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0454003520 con 0x7f046815ddc0 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046e2e5640 1 --2- 172.21.15.12:0/133123074 >> v2:172.21.15.156:3300/0 conn(0x7f04680b8300 0x7f04681688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115187744 0 0) 0x7f0454003520 con 0x7f046815ddc0 2024-03-08T08:29:05.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 >> v2:172.21.15.156:3300/0 conn(0x7f04680b8300 msgr2=0x7f04681688d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 --2- 172.21.15.12:0/133123074 >> v2:172.21.15.156:3300/0 conn(0x7f04680b8300 0x7f04681688d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 >> v1:172.21.15.156:6789/0 conn(0x7f04680d1110 legacy=0x7f04680b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680b8a70 con 0x7f046815ddc0 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f0464001670 con 0x7f046815ddc0 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 >> v1:172.21.15.12:6789/0 conn(0x7f046815ddc0 legacy=0x7f04680d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04540039b0 0x7f0454003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0454004800 0x7f0454004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f045400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045400c030 con 0x7f04540051c0 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045400c1e0 con 0x7f0454004800 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.366+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045400c3e0 con 0x7f04540039b0 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f045400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046d2e3640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0454004800 0x7f0454004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046dae4640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04540039b0 0x7f0454003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 -- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04540039b0 msgr2=0x7f0454003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04540039b0 0x7f0454003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 -- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0454004800 msgr2=0x7f0454004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0454004800 0x7f0454004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 -- 172.21.15.12:0/133123074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0454003520 con 0x7f04540051c0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.367+0000 7f046e2e5640 1 -- 172.21.15.12:0/133123074 reap_dead start 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046e2e5640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f045400b9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f90 tx=0x7f0454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeaf99197a560ca server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460039070 con 0x7f04540051c0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046002cc20 con 0x7f04540051c0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046cae2640 1 -- 172.21.15.12:0/133123074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460009c10 con 0x7f04540051c0 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 -- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 msgr2=0x7f045400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f045400b9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f90 tx=0x7f0454005830 comp rx=0 tx=0).stop 2024-03-08T08:29:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 -- 172.21.15.12:0/133123074 shutdown_connections 2024-03-08T08:29:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 --2- 172.21.15.12:0/133123074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f045400b9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 -- 172.21.15.12:0/133123074 >> 172.21.15.12:0/133123074 conn(0x7f0468072720 msgr2=0x7f0468072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 -- 172.21.15.12:0/133123074 shutdown_connections 2024-03-08T08:29:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.368+0000 7f046fd48640 1 -- 172.21.15.12:0/133123074 wait complete. 2024-03-08T08:29:05.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.369+0000 7f046fd48640 1 Processor -- start 2024-03-08T08:29:05.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.369+0000 7f046fd48640 1 -- start start 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 0x7f04681f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 0x7f04681f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680b9980 con 0x7f04540051c0 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680d1d20 con 0x7f04681f3ca0 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046fd48640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04680d0a00 con 0x7f04680aae80 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53994/0 (socket says 172.21.15.12:53994) 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 -- 172.21.15.12:0/1377449943 learned_addr learned my addr 172.21.15.12:0/1377449943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046e2e5640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 0x7f04681f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046d2e3640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 0x7f04681f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 -- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 msgr2=0x7f04681f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 0x7f04681f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 -- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 msgr2=0x7f04681f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 0x7f04681f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.370+0000 7f046dae4640 1 -- 172.21.15.12:0/1377449943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680ab6b0 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f046e2e5640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 0x7f04681f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f046d2e3640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 0x7f04681f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f046dae4640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04640072c0 tx=0x7f0464007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400f8e0 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0464018070 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04680ab990 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400fce0 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.371+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04680abf50 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.372+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04680749c0 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.372+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f046401d460 con 0x7f04540051c0 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.372+0000 7f045e7fc640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:05.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.373+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f046408a240 con 0x7f04540051c0 2024-03-08T08:29:05.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.373+0000 7f046d2e3640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:05.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.373+0000 7f046d2e3640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 0x7f044c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04680b7d90 tx=0x7f0458009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:05.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.377+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0464059120 con 0x7f04540051c0 2024-03-08T08:29:05.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:05.595+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f04680d42a0 con 0x7f04540051c0 2024-03-08T08:29:06.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.230+0000 7f045e7fc640 1 -- 172.21.15.12:0/1377449943 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0464058a20 con 0x7f04540051c0 2024-03-08T08:29:06.232 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-03-08T08:29:06.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.236+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 msgr2=0x7f044c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.236+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 0x7f044c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04680b7d90 tx=0x7f0458009040 comp rx=0 tx=0).stop 2024-03-08T08:29:06.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.236+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 msgr2=0x7f04680aa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.236+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04640072c0 tx=0x7f0464007780 comp rx=0 tx=0).stop 2024-03-08T08:29:06.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 shutdown_connections 2024-03-08T08:29:06.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04681f3ca0 0x7f04681f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04680aae80 0x7f04681f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 --2- 172.21.15.12:0/1377449943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04540051c0 0x7f04680aa940 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.237+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 >> 172.21.15.12:0/1377449943 conn(0x7f0468072720 msgr2=0x7f04680b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.258+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 shutdown_connections 2024-03-08T08:29:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.259+0000 7f046fd48640 1 -- 172.21.15.12:0/1377449943 wait complete. 2024-03-08T08:29:06.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.379+0000 7fe567fc7640 1 Processor -- start 2024-03-08T08:29:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.379+0000 7fe567fc7640 1 -- start start 2024-03-08T08:29:06.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe567fc7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe5600d0a30 0x7fe5600b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe567fc7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe560161fb0 con 0x7fe560161840 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe567fc7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe560162290 con 0x7fe56015df10 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe567fc7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600d1d60 con 0x7fe5600d0a30 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe566564640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe560161840 0x7fe5600d24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44980/0 (socket says 172.21.15.12:44980) 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe566564640 1 -- 172.21.15.12:0/2375755179 learned_addr learned my addr 172.21.15.12:0/2375755179 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722709263 0 0) 0x7fe560161fb0 con 0x7fe560161840 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54c003520 con 0x7fe560161840 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe565d63640 1 --2- 172.21.15.12:0/2375755179 >> v2:172.21.15.156:3300/0 conn(0x7fe5600d0a30 0x7fe5600b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1853642531 0 0) 0x7fe54c003520 con 0x7fe560161840 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 >> v2:172.21.15.156:3300/0 conn(0x7fe5600d0a30 msgr2=0x7fe5600b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 --2- 172.21.15.12:0/2375755179 >> v2:172.21.15.156:3300/0 conn(0x7fe5600d0a30 0x7fe5600b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 >> v1:172.21.15.156:6789/0 conn(0x7fe56015df10 legacy=0x7fe5600d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560162570 con 0x7fe560161840 2024-03-08T08:29:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fe5580019f0 con 0x7fe560161840 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 >> v1:172.21.15.12:6789/0 conn(0x7fe560161840 legacy=0x7fe5600d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c0039b0 0x7fe54c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c004800 0x7fe54c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 0x7fe54c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600d1d60 con 0x7fe54c0051c0 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c00c110 con 0x7fe54c0039b0 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c00c310 con 0x7fe54c004800 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe566564640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 0x7fe54c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.380+0000 7fe565d63640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c0039b0 0x7fe54c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe565562640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c004800 0x7fe54c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 -- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c004800 msgr2=0x7fe54c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c004800 0x7fe54c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 -- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c0039b0 msgr2=0x7fe54c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c0039b0 0x7fe54c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 -- 172.21.15.12:0/2375755179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c003520 con 0x7fe54c0051c0 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe565562640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c004800 0x7fe54c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 -- 172.21.15.12:0/2375755179 reap_dead start 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.381+0000 7fe566564640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 0x7fe54c00b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe55800a860 tx=0x7fe54c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1a4aa649ce2bad server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800a480 con 0x7fe54c0051c0 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe558034630 con 0x7fe54c0051c0 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe564d61640 1 -- 172.21.15.12:0/2375755179 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558034a20 con 0x7fe54c0051c0 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 -- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 msgr2=0x7fe54c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 0x7fe54c00b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe55800a860 tx=0x7fe54c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 -- 172.21.15.12:0/2375755179 shutdown_connections 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 --2- 172.21.15.12:0/2375755179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe54c0051c0 0x7fe54c00b9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 -- 172.21.15.12:0/2375755179 >> 172.21.15.12:0/2375755179 conn(0x7fe56006c710 msgr2=0x7fe56006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:06.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 -- 172.21.15.12:0/2375755179 shutdown_connections 2024-03-08T08:29:06.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.382+0000 7fe567fc7640 1 -- 172.21.15.12:0/2375755179 wait complete. 2024-03-08T08:29:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.383+0000 7fe567fc7640 1 Processor -- start 2024-03-08T08:29:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.383+0000 7fe567fc7640 1 -- start start 2024-03-08T08:29:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c0051c0 0x7fe5600cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5601f4320 0x7fe5601f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600b95a0 con 0x7fe56015ae50 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560161cb0 con 0x7fe5601f4320 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe567fc7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600b9720 con 0x7fe54c0051c0 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54008/0 (socket says 172.21.15.12:54008) 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 -- 172.21.15.12:0/1120006263 learned_addr learned my addr 172.21.15.12:0/1120006263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe566564640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5601f4320 0x7fe5601f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565d63640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c0051c0 0x7fe5600cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 -- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c0051c0 msgr2=0x7fe5600cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c0051c0 0x7fe5600cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 -- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5601f4320 msgr2=0x7fe5601f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5601f4320 0x7fe5601f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.384+0000 7fe565562640 1 -- 172.21.15.12:0/1120006263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5601f73c0 con 0x7fe56015ae50 2024-03-08T08:29:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe565562640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe55000e480 tx=0x7fe55000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5500052e0 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe550018070 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550003830 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5601f7680 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.385+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5601f7c40 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.386+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5600749c0 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.386+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5500039d0 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.387+0000 7fe5567fc640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 0x7fe54406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.387+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe550089c70 con 0x7fe56015ae50 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.387+0000 7fe565d63640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 0x7fe54406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.387+0000 7fe565d63640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 0x7fe54406d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe55c006fd0 tx=0x7fe55c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.391+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe550058530 con 0x7fe56015ae50 2024-03-08T08:29:06.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.573+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5600d1630 con 0x7fe56015ae50 2024-03-08T08:29:06.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.574+0000 7fe5567fc640 1 -- 172.21.15.12:0/1120006263 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16622 (secure 0 0 0) 0x7fe55005c5d0 con 0x7fe56015ae50 2024-03-08T08:29:06.576 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:06.576 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:05.596271+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 msgr2=0x7fe54406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 0x7fe54406d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe55c006fd0 tx=0x7fe55c008040 comp rx=0 tx=0).stop 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 msgr2=0x7fe56015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe55000e480 tx=0x7fe55000e940 comp rx=0 tx=0).stop 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 shutdown_connections 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe54406b500 0x7fe54406d9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5601f4320 0x7fe5601f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe56015ae50 0x7fe56015b2b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 --2- 172.21.15.12:0/1120006263 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe54c0051c0 0x7fe5600cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 >> 172.21.15.12:0/1120006263 conn(0x7fe56006c710 msgr2=0x7fe5600c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.576+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 shutdown_connections 2024-03-08T08:29:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.577+0000 7fe567fc7640 1 -- 172.21.15.12:0/1120006263 wait complete. 2024-03-08T08:29:06.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.689+0000 7f5abd415640 1 Processor -- start 2024-03-08T08:29:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.689+0000 7f5abd415640 1 -- start start 2024-03-08T08:29:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5abd415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5ab80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5abd415640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80d0910 con 0x7f5ab809b150 2024-03-08T08:29:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5abd415640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80cca40 con 0x7f5ab815dff0 2024-03-08T08:29:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5abd415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80ccd20 con 0x7f5ab809a3b0 2024-03-08T08:29:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5aaffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5ab809b150 0x7f5ab8099ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44996/0 (socket says 172.21.15.12:44996) 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5aaffff640 1 -- 172.21.15.12:0/3267104982 learned_addr learned my addr 172.21.15.12:0/3267104982 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757808267 0 0) 0x7f5ab80d0910 con 0x7f5ab809b150 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa4003520 con 0x7f5ab809b150 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab77fe640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5ab80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422719462 0 0) 0x7f5aa4003520 con 0x7f5ab809b150 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 msgr2=0x7f5ab80cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5ab80cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 >> v1:172.21.15.156:6789/0 conn(0x7f5ab815dff0 legacy=0x7f5ab80cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80cc6f0 con 0x7f5ab809b150 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f5aa0001670 con 0x7f5ab809b150 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 >> v1:172.21.15.12:6789/0 conn(0x7f5ab809b150 legacy=0x7f5ab8099ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5aa4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa4004800 0x7f5aa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5aa4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80ccd20 con 0x7f5aa40039e0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa400bb20 con 0x7f5aa4004800 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa400bd20 con 0x7f5ab809a3b0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5aa4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5aaffff640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa4004800 0x7f5aa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.690+0000 7f5ab77fe640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5aa4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 msgr2=0x7f5aa4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5aa4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa4004800 msgr2=0x7f5aa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa4004800 0x7f5aa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/3267104982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4003520 con 0x7f5aa40039e0 2024-03-08T08:29:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab77fe640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab809a3b0 0x7f5aa4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5aa4003e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8001750 tx=0x7f5aa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c521a561b35e0d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8001370 con 0x7f5aa40039e0 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa80052d0 con 0x7f5aa40039e0 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.691+0000 7f5ab67fc640 1 -- 172.21.15.12:0/3267104982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8003830 con 0x7f5aa40039e0 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 -- 172.21.15.12:0/3267104982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 msgr2=0x7f5aa4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 --2- 172.21.15.12:0/3267104982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5aa4003e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8001750 tx=0x7f5aa4005830 comp rx=0 tx=0).stop 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5ab77fe640 1 -- 172.21.15.12:0/3267104982 reap_dead start 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 -- 172.21.15.12:0/3267104982 shutdown_connections 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 -- 172.21.15.12:0/3267104982 >> 172.21.15.12:0/3267104982 conn(0x7f5ab80d1900 msgr2=0x7f5ab80d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 -- 172.21.15.12:0/3267104982 shutdown_connections 2024-03-08T08:29:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.692+0000 7f5abd415640 1 -- 172.21.15.12:0/3267104982 wait complete. 2024-03-08T08:29:06.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.693+0000 7f5abd415640 1 Processor -- start 2024-03-08T08:29:06.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.693+0000 7f5abd415640 1 -- start start 2024-03-08T08:29:06.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab80b4520 0x7f5ab8153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 0x7f5ab80b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cf650 con 0x7f5aa40039e0 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cf4d0 con 0x7f5ab80b4d30 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5abd415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cf7d0 con 0x7f5ab80b4520 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54032/0 (socket says 172.21.15.12:54032) 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/4007857735 learned_addr learned my addr 172.21.15.12:0/4007857735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab77fe640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 0x7f5ab80b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5aaffff640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab80b4520 0x7f5ab8153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab80b4520 msgr2=0x7f5ab8153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab80b4520 0x7f5ab8153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 msgr2=0x7f5ab80b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 0x7f5ab80b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab6ffd640 1 -- 172.21.15.12:0/4007857735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80b4980 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.694+0000 7f5ab77fe640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 0x7f5ab80b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5ab6ffd640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8005040 tx=0x7f5aa8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8041070 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa8005480 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab815d330 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8001370 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.695+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab80c9b70 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.696+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab80cd420 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.696+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5aa8044070 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.696+0000 7f5aaf7fe640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.697+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5aa80abc60 con 0x7f5aa40039e0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.697+0000 7f5aaffff640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:06.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.697+0000 7f5aaffff640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 0x7f5a9806d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80b7750 tx=0x7f5aa0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:06.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.701+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ab80cd420 con 0x7f5aa40039e0 2024-03-08T08:29:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.882+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab80749c0 con 0x7f5aa40039e0 2024-03-08T08:29:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.883+0000 7f5aaf7fe640 1 -- 172.21.15.12:0/4007857735 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16622 (secure 0 0 0) 0x7f5aa807aae0 con 0x7f5aa40039e0 2024-03-08T08:29:06.884 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:06.884 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:05.596271+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 msgr2=0x7f5a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 0x7f5a9806d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80b7750 tx=0x7f5aa0009040 comp rx=0 tx=0).stop 2024-03-08T08:29:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 msgr2=0x7f5ab80b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8005040 tx=0x7f5aa8001120 comp rx=0 tx=0).stop 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 shutdown_connections 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ab80b4d30 0x7f5ab80b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ab80b4520 0x7f5ab8153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 --2- 172.21.15.12:0/4007857735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aa40039e0 0x7f5ab80b6e40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 >> 172.21.15.12:0/4007857735 conn(0x7f5ab80d1900 msgr2=0x7f5ab80cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 shutdown_connections 2024-03-08T08:29:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:06.885+0000 7f5abd415640 1 -- 172.21.15.12:0/4007857735 wait complete. 2024-03-08T08:29:06.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-08T08:29:07.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.000+0000 7f176c381640 1 Processor -- start 2024-03-08T08:29:07.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.000+0000 7f176c381640 1 -- start start 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176c381640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17640b9760 0x7f17640d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176c381640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640b7250 con 0x7f176415dbc0 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176c381640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640b7520 con 0x7f17640d2050 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176c381640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17640b7800 con 0x7f17640b9760 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176991c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17640b9760 0x7f17640d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176a11d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f176415dbc0 0x7f17640b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45008/0 (socket says 172.21.15.12:45008) 2024-03-08T08:29:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176991c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17640b9760 0x7f17640d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54034/0 (socket says 172.21.15.12:54034) 2024-03-08T08:29:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.001+0000 7f176a11d640 1 -- 172.21.15.12:0/3277634696 learned_addr learned my addr 172.21.15.12:0/3277634696 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.002+0000 7f176991c640 1 -- 172.21.15.12:0/3277634696 >> v1:172.21.15.156:6789/0 conn(0x7f17640d2050 legacy=0x7f17640b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.002+0000 7f176991c640 1 -- 172.21.15.12:0/3277634696 >> v1:172.21.15.12:6789/0 conn(0x7f176415dbc0 legacy=0x7f17640b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.002+0000 7f176911b640 1 -- 172.21.15.12:0/3277634696 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673672704 0 0) 0x7f17640b7250 con 0x7f176415dbc0 2024-03-08T08:29:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.002+0000 7f176991c640 1 -- 172.21.15.12:0/3277634696 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640b7ae0 con 0x7f17640b9760 2024-03-08T08:29:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.002+0000 7f176991c640 1 --2- 172.21.15.12:0/3277634696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17640b9760 0x7f17640d1b10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17580097a0 tx=0x7f175802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67993088561f51d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176911b640 1 -- 172.21.15.12:0/3277634696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175802f810 con 0x7f17640b9760 2024-03-08T08:29:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176911b640 1 -- 172.21.15.12:0/3277634696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175802f9b0 con 0x7f17640b9760 2024-03-08T08:29:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176911b640 1 -- 172.21.15.12:0/3277634696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175802fd80 con 0x7f17640b9760 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 -- 172.21.15.12:0/3277634696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17640b9760 msgr2=0x7f17640d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 --2- 172.21.15.12:0/3277634696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17640b9760 0x7f17640d1b10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17580097a0 tx=0x7f175802b9f0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 -- 172.21.15.12:0/3277634696 shutdown_connections 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 --2- 172.21.15.12:0/3277634696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17640b9760 0x7f17640d1b10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 -- 172.21.15.12:0/3277634696 >> 172.21.15.12:0/3277634696 conn(0x7f1764072720 msgr2=0x7f1764072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 -- 172.21.15.12:0/3277634696 shutdown_connections 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.003+0000 7f176c381640 1 -- 172.21.15.12:0/3277634696 wait complete. 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.004+0000 7f176c381640 1 Processor -- start 2024-03-08T08:29:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.004+0000 7f176c381640 1 -- start start 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17640b9760 0x7f17640cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 0x7f17640ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176415dfa0 con 0x7f176415dbc0 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640b80a0 con 0x7f17640d2050 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176c381640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17640b8cd0 con 0x7f17640b9760 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a91e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a91e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54036/0 (socket says 172.21.15.12:54036) 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a91e640 1 -- 172.21.15.12:0/3953492153 learned_addr learned my addr 172.21.15.12:0/3953492153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176991c640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 0x7f17640ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a91e640 1 -- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17640b9760 msgr2=0x7f17640cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a11d640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17640b9760 0x7f17640cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.005+0000 7f176a91e640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17640b9760 0x7f17640cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176a91e640 1 -- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 msgr2=0x7f17640ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176a91e640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 0x7f17640ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176a91e640 1 -- 172.21.15.12:0/3953492153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640abc80 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176991c640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 0x7f17640ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176a91e640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f175400c7a0 tx=0x7f175400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754003b70 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1754003d30 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400aa90 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17640cc040 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.006+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17640cc600 con 0x7f176415dbc0 2024-03-08T08:29:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.007+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17640749c0 con 0x7f176415dbc0 2024-03-08T08:29:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.007+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f175400ac30 con 0x7f176415dbc0 2024-03-08T08:29:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.008+0000 7f1762ffd640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 0x7f175006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.008+0000 7f176a11d640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 0x7f175006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.008+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f175408a150 con 0x7f176415dbc0 2024-03-08T08:29:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.008+0000 7f176a11d640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 0x7f175006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f175c000f60 tx=0x7f175c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.012+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1754058fd0 con 0x7f176415dbc0 2024-03-08T08:29:07.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.230+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f176406b200 con 0x7f176415dbc0 2024-03-08T08:29:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.251+0000 7f1762ffd640 1 -- 172.21.15.12:0/3953492153 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f17540588d0 con 0x7f176415dbc0 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 msgr2=0x7f175006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 0x7f175006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f175c000f60 tx=0x7f175c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 msgr2=0x7f17640ab740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f175400c7a0 tx=0x7f175400cc60 comp rx=0 tx=0).stop 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 shutdown_connections 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f175006b4d0 0x7f175006d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f176415dbc0 0x7f17640ab740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17640d2050 0x7f17640ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 --2- 172.21.15.12:0/3953492153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17640b9760 0x7f17640cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 >> 172.21.15.12:0/3953492153 conn(0x7f1764072720 msgr2=0x7f17640d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.253+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 shutdown_connections 2024-03-08T08:29:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.254+0000 7f176c381640 1 -- 172.21.15.12:0/3953492153 wait complete. 2024-03-08T08:29:07.265 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-08T08:29:07.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-08T08:29:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.368+0000 7f30ae5d7640 1 Processor -- start 2024-03-08T08:29:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.368+0000 7f30ae5d7640 1 -- start start 2024-03-08T08:29:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30ae5d7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f30a80b9760 0x7f30a80d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30ae5d7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30a80d2050 0x7f30a80b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30ae5d7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80b7250 con 0x7f30a815dbc0 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30ae5d7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80b7520 con 0x7f30a80d2050 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30ae5d7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80b7800 con 0x7f30a80b9760 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30acb74640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30a80d2050 0x7f30a80b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30a7fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f30a815dbc0 0x7f30a80b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45020/0 (socket says 172.21.15.12:45020) 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30a7fff640 1 -- 172.21.15.12:0/1381298335 learned_addr learned my addr 172.21.15.12:0/1381298335 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30a77fe640 1 --2- 172.21.15.12:0/1381298335 >> v2:172.21.15.156:3300/0 conn(0x7f30a80b9760 0x7f30a80d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30acb74640 1 -- 172.21.15.12:0/1381298335 >> v2:172.21.15.156:3300/0 conn(0x7f30a80b9760 msgr2=0x7f30a80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30acb74640 1 --2- 172.21.15.12:0/1381298335 >> v2:172.21.15.156:3300/0 conn(0x7f30a80b9760 0x7f30a80d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30acb74640 1 -- 172.21.15.12:0/1381298335 >> v1:172.21.15.12:6789/0 conn(0x7f30a815dbc0 legacy=0x7f30a80b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30acb74640 1 -- 172.21.15.12:0/1381298335 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a80b7ae0 con 0x7f30a80d2050 2024-03-08T08:29:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.369+0000 7f30a6ffd640 1 -- 172.21.15.12:0/1381298335 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336779686 0 0) 0x7f30a80b7250 con 0x7f30a815dbc0 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.370+0000 7f30acb74640 1 --2- 172.21.15.12:0/1381298335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80d2050 0x7f30a80b85c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f30a80cab30 tx=0x7f30a002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518fb20c1ba6f5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.370+0000 7f30a6ffd640 1 -- 172.21.15.12:0/1381298335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0039070 con 0x7f30a80d2050 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.370+0000 7f30a6ffd640 1 -- 172.21.15.12:0/1381298335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a002cd00 con 0x7f30a80d2050 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.370+0000 7f30a6ffd640 1 -- 172.21.15.12:0/1381298335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0034660 con 0x7f30a80d2050 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 -- 172.21.15.12:0/1381298335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80d2050 msgr2=0x7f30a80b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/1381298335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80d2050 0x7f30a80b85c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f30a80cab30 tx=0x7f30a002bf50 comp rx=0 tx=0).stop 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 -- 172.21.15.12:0/1381298335 shutdown_connections 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/1381298335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80d2050 0x7f30a80b85c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/1381298335 >> v2:172.21.15.156:3300/0 conn(0x7f30a80b9760 0x7f30a80d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 -- 172.21.15.12:0/1381298335 >> 172.21.15.12:0/1381298335 conn(0x7f30a8072720 msgr2=0x7f30a8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 -- 172.21.15.12:0/1381298335 shutdown_connections 2024-03-08T08:29:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.371+0000 7f30ae5d7640 1 -- 172.21.15.12:0/1381298335 wait complete. 2024-03-08T08:29:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.372+0000 7f30ae5d7640 1 Processor -- start 2024-03-08T08:29:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.372+0000 7f30ae5d7640 1 -- start start 2024-03-08T08:29:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 0x7f30a8159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a815dbc0 0x7f30a8158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a815dfa0 con 0x7f30a80b9760 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80b80a0 con 0x7f30a80d2050 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80b8b00 con 0x7f30a815dbc0 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54062/0 (socket says 172.21.15.12:54062) 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 -- 172.21.15.12:0/2048543283 learned_addr learned my addr 172.21.15.12:0/2048543283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a77fe640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 0x7f30a8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 -- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a815dbc0 msgr2=0x7f30a8158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a815dbc0 0x7f30a8158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 -- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 msgr2=0x7f30a8159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 0x7f30a8159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 -- 172.21.15.12:0/2048543283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a81597a0 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a77fe640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 0x7f30a8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a7fff640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c430 tx=0x7f309c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c0074e0 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c0047e0 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a8159a80 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c004bd0 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.373+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a815a040 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.374+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a80749c0 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.375+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f309c011070 con 0x7f30a80b9760 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.375+0000 7f30a4ff9640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 0x7f309406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.375+0000 7f30a77fe640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 0x7f309406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.375+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f309c089e30 con 0x7f30a80b9760 2024-03-08T08:29:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.376+0000 7f30a77fe640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 0x7f309406d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f30a80d0ba0 tx=0x7f3098008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.380+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f309c058d40 con 0x7f30a80b9760 2024-03-08T08:29:07.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:07.597+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f30a80b5f60 con 0x7f30a80b9760 2024-03-08T08:29:08.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.257+0000 7f30a4ff9640 1 -- 172.21.15.12:0/2048543283 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f309c058640 con 0x7f30a80b9760 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 msgr2=0x7f309406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 0x7f309406d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f30a80d0ba0 tx=0x7f3098008040 comp rx=0 tx=0).stop 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 msgr2=0x7f30a815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c430 tx=0x7f309c00c8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 shutdown_connections 2024-03-08T08:29:08.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f309406b4d0 0x7f309406d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a815dbc0 0x7f30a8158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80d2050 0x7f30a8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 --2- 172.21.15.12:0/2048543283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80b9760 0x7f30a815a840 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 >> 172.21.15.12:0/2048543283 conn(0x7f30a8072720 msgr2=0x7f30a815d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 shutdown_connections 2024-03-08T08:29:08.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.260+0000 7f30ae5d7640 1 -- 172.21.15.12:0/2048543283 wait complete. 2024-03-08T08:29:08.274 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-08T08:29:08.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.379+0000 7f8800381640 1 Processor -- start 2024-03-08T08:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.379+0000 7f8800381640 1 -- start start 2024-03-08T08:29:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f8800381640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f87f80d1110 0x7f87f80b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f8800381640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f815e1a0 con 0x7f87f80d31c0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f8800381640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80b8790 con 0x7f87f815ddc0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f8800381640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f80b9030 con 0x7f87f80d1110 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fe91e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f87f80d31c0 0x7f87f80b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45032/0 (socket says 172.21.15.12:45032) 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fe91e640 1 -- 172.21.15.12:0/45971824 learned_addr learned my addr 172.21.15.12:0/45971824 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd91c640 1 --2- 172.21.15.12:0/45971824 >> v2:172.21.15.156:3300/0 conn(0x7f87f80d1110 0x7f87f80b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794159451 0 0) 0x7f87f815e1a0 con 0x7f87f80d31c0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e4003520 con 0x7f87f80d31c0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811601123 0 0) 0x7f87e4003520 con 0x7f87f80d31c0 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 >> v2:172.21.15.156:3300/0 conn(0x7f87f80d1110 msgr2=0x7f87f80b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 --2- 172.21.15.12:0/45971824 >> v2:172.21.15.156:3300/0 conn(0x7f87f80d1110 0x7f87f80b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 >> v1:172.21.15.156:6789/0 conn(0x7f87f815ddc0 legacy=0x7f87f80d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f80b8a70 con 0x7f87f80d31c0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.380+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f87f00019f0 con 0x7f87f80d31c0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 >> v1:172.21.15.12:6789/0 conn(0x7f87f80d31c0 legacy=0x7f87f80b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40039b0 0x7f87e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87e4004800 0x7f87e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 0x7f87e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e400c030 con 0x7f87e40051c0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e400c1e0 con 0x7f87e40039b0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e400c3e0 con 0x7f87e4004800 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 0x7f87e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe11d640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40039b0 0x7f87e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fd91c640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87e4004800 0x7f87e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 -- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87e4004800 msgr2=0x7f87e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87e4004800 0x7f87e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 -- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40039b0 msgr2=0x7f87e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40039b0 0x7f87e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.381+0000 7f87fe91e640 1 -- 172.21.15.12:0/45971824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4003520 con 0x7f87e40051c0 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f87fe91e640 1 -- 172.21.15.12:0/45971824 reap_dead start 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f87fe91e640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 0x7f87e400b9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87f000a860 tx=0x7f87e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af88b1a1b06bbe6d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f000a480 con 0x7f87e40051c0 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0034630 con 0x7f87e40051c0 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f87fd11b640 1 -- 172.21.15.12:0/45971824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0034a20 con 0x7f87e40051c0 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f8800381640 1 -- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 msgr2=0x7f87e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f8800381640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 0x7f87e400b9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87f000a860 tx=0x7f87e4005830 comp rx=0 tx=0).stop 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f8800381640 1 -- 172.21.15.12:0/45971824 shutdown_connections 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f8800381640 1 --2- 172.21.15.12:0/45971824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87e40051c0 0x7f87e400b9c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.382+0000 7f8800381640 1 -- 172.21.15.12:0/45971824 >> 172.21.15.12:0/45971824 conn(0x7f87f8072720 msgr2=0x7f87f8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.383+0000 7f8800381640 1 -- 172.21.15.12:0/45971824 shutdown_connections 2024-03-08T08:29:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.383+0000 7f8800381640 1 -- 172.21.15.12:0/45971824 wait complete. 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 Processor -- start 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 -- start start 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40051c0 0x7f87f81f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87f815f1f0 0x7f87f8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80b9980 con 0x7f87f81f65f0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80d1d20 con 0x7f87e40051c0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.384+0000 7f8800381640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80d0a00 con 0x7f87f815f1f0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54074/0 (socket says 172.21.15.12:54074) 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 -- 172.21.15.12:0/2902928518 learned_addr learned my addr 172.21.15.12:0/2902928518 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe11d640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40051c0 0x7f87f81f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fd91c640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87f815f1f0 0x7f87f8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 -- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87f815f1f0 msgr2=0x7f87f8154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87f815f1f0 0x7f87f8154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 -- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40051c0 msgr2=0x7f87f81f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40051c0 0x7f87f81f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 -- 172.21.15.12:0/2902928518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f815f760 con 0x7f87f81f65f0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87fe91e640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f87f002cc70 tx=0x7f87f80d1c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f000a480 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f00019c0 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0034430 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f815fa40 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.385+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f8160000 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.386+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f80749c0 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.387+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f0046020 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.387+0000 7f87eeffd640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 0x7f87dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.387+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87f00ab3b0 con 0x7f87f81f65f0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.387+0000 7f87fe11d640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 0x7f87dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.387+0000 7f87fe11d640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 0x7f87dc06d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87f4000c00 tx=0x7f87f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.391+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f87f004d020 con 0x7f87f81f65f0 2024-03-08T08:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.609+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f815ca70 con 0x7f87f81f65f0 2024-03-08T08:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.609+0000 7f87eeffd640 1 -- 172.21.15.12:0/2902928518 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f87f007a230 con 0x7f87f81f65f0 2024-03-08T08:29:08.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:08.611 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 msgr2=0x7f87dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 0x7f87dc06d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87f4000c00 tx=0x7f87f4009040 comp rx=0 tx=0).stop 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 msgr2=0x7f87f81f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f87f002cc70 tx=0x7f87f80d1c90 comp rx=0 tx=0).stop 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 shutdown_connections 2024-03-08T08:29:08.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87dc06b500 0x7f87dc06d9c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81f65f0 0x7f87f81f6a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87f815f1f0 0x7f87f8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 --2- 172.21.15.12:0/2902928518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87e40051c0 0x7f87f81f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 >> 172.21.15.12:0/2902928518 conn(0x7f87f8072720 msgr2=0x7f87f80ce510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 shutdown_connections 2024-03-08T08:29:08.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.612+0000 7f8800381640 1 -- 172.21.15.12:0/2902928518 wait complete. 2024-03-08T08:29:08.623 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.723+0000 7f0e3d1fd640 1 Processor -- start 2024-03-08T08:29:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.723+0000 7f0e3d1fd640 1 -- start start 2024-03-08T08:29:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3d1fd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3d1fd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 0x7f0e380b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3d1fd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3815e1a0 con 0x7f0e3815ddc0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3d1fd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e380b9030 con 0x7f0e380d31c0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3d1fd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380d1d20 con 0x7f0e380d1110 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e36d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0e3815ddc0 0x7f0e380d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52752/0 (socket says 172.21.15.12:52752) 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e36d9d640 1 -- 172.21.15.12:0/1344497339 learned_addr learned my addr 172.21.15.12:0/1344497339 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3759e640 1 --2- 172.21.15.12:0/1344497339 >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 0x7f0e380b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3659c640 1 --2- 172.21.15.12:0/1344497339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272632724 0 0) 0x7f0e3815e1a0 con 0x7f0e3815ddc0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24003520 con 0x7f0e3815ddc0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668112655 0 0) 0x7f0e24003520 con 0x7f0e3815ddc0 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3659c640 1 -- 172.21.15.12:0/1344497339 >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 msgr2=0x7f0e380b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3659c640 1 --2- 172.21.15.12:0/1344497339 >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 0x7f0e380b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3659c640 1 -- 172.21.15.12:0/1344497339 >> v1:172.21.15.12:6789/0 conn(0x7f0e3815ddc0 legacy=0x7f0e380d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3659c640 1 -- 172.21.15.12:0/1344497339 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e380b8790 con 0x7f0e380d1110 2024-03-08T08:29:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.724+0000 7f0e3759e640 1 --2- 172.21.15.12:0/1344497339 >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 0x7f0e380b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.725+0000 7f0e3659c640 1 --2- 172.21.15.12:0/1344497339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380b6be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e200097a0 tx=0x7f0e2002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf2d1d9a996070c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.725+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20037070 con 0x7f0e380d1110 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.725+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e20030510 con 0x7f0e380d1110 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.725+0000 7f0e35d9b640 1 -- 172.21.15.12:0/1344497339 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e30001670 con 0x7f0e380d1110 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/1344497339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 msgr2=0x7f0e380b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/1344497339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380b6be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e200097a0 tx=0x7f0e2002b790 comp rx=0 tx=0).stop 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/1344497339 shutdown_connections 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/1344497339 >> v2:172.21.15.156:3300/0 conn(0x7f0e380d31c0 0x7f0e380b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/1344497339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380b6be0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/1344497339 >> 172.21.15.12:0/1344497339 conn(0x7f0e38072720 msgr2=0x7f0e38072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/1344497339 shutdown_connections 2024-03-08T08:29:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.726+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/1344497339 wait complete. 2024-03-08T08:29:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.727+0000 7f0e3d1fd640 1 Processor -- start 2024-03-08T08:29:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.727+0000 7f0e3d1fd640 1 -- start start 2024-03-08T08:29:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 0x7f0e3815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380b9980 con 0x7f0e380d31c0 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380b8d50 con 0x7f0e3815ddc0 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3d1fd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380b7660 con 0x7f0e380d1110 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57658/0 (socket says 172.21.15.12:57658) 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3659c640 1 -- 172.21.15.12:0/3776658916 learned_addr learned my addr 172.21.15.12:0/3776658916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e3759e640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 0x7f0e3815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.728+0000 7f0e36d9d640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 -- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 msgr2=0x7f0e380c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 -- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 msgr2=0x7f0e3815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 0x7f0e3815eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 -- 172.21.15.12:0/3776658916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3815fc60 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3759e640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 0x7f0e3815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3659c640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e2002b7d0 tx=0x7f0e200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20040020 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e20038040 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3815ff40 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20037070 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.729+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e38160500 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.730+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e380749c0 con 0x7f0e380d31c0 2024-03-08T08:29:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.730+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e20043020 con 0x7f0e380d31c0 2024-03-08T08:29:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.731+0000 7f0e2f7fe640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 0x7f0e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.731+0000 7f0e36d9d640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 0x7f0e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.731+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e200ab660 con 0x7f0e380d31c0 2024-03-08T08:29:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.731+0000 7f0e36d9d640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 0x7f0e1806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e3806c6b0 tx=0x7f0e30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.735+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0e2004a030 con 0x7f0e380d31c0 2024-03-08T08:29:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.918+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e380b2740 con 0x7f0e380d31c0 2024-03-08T08:29:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.919+0000 7f0e2f7fe640 1 -- 172.21.15.12:0/3776658916 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f0e2007a4e0 con 0x7f0e380d31c0 2024-03-08T08:29:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:08.920 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 msgr2=0x7f0e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 0x7f0e1806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e3806c6b0 tx=0x7f0e30009040 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 msgr2=0x7f0e38160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e2002b7d0 tx=0x7f0e200046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 shutdown_connections 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e1806b4d0 0x7f0e1806d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3815ddc0 0x7f0e3815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e380d31c0 0x7f0e38160d00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 --2- 172.21.15.12:0/3776658916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e380d1110 0x7f0e380c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 >> 172.21.15.12:0/3776658916 conn(0x7f0e38072720 msgr2=0x7f0e3815d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 shutdown_connections 2024-03-08T08:29:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:08.921+0000 7f0e3d1fd640 1 -- 172.21.15.12:0/3776658916 wait complete. 2024-03-08T08:29:08.933 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-08T08:29:08.933 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 Processor -- start 2024-03-08T08:29:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 -- start start 2024-03-08T08:29:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cc80b8300 0x7f1cc81688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc815e1a0 con 0x7f1cc80d1110 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80b8790 con 0x7f1cc815ddc0 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.035+0000 7f1cce2a7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80b9030 con 0x7f1cc80b8300 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cc80b8300 0x7f1cc81688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cc80b8300 0x7f1cc81688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57662/0 (socket says 172.21.15.12:57662) 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 -- 172.21.15.12:0/148199610 learned_addr learned my addr 172.21.15.12:0/148199610 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1cc77fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1cc80d1110 0x7f1cc80b6be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52758/0 (socket says 172.21.15.12:52758) 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1cc7fff640 1 --1- 172.21.15.12:0/148199610 >> v1:172.21.15.156:6789/0 conn(0x7f1cc815ddc0 0x7f1cc80d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47012/0 (socket says 172.21.15.12:47012) 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 -- 172.21.15.12:0/148199610 >> v1:172.21.15.156:6789/0 conn(0x7f1cc815ddc0 legacy=0x7f1cc80d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 -- 172.21.15.12:0/148199610 >> v1:172.21.15.12:6789/0 conn(0x7f1cc80d1110 legacy=0x7f1cc80b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1ccc844640 1 -- 172.21.15.12:0/148199610 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80b8a70 con 0x7f1cc80b8300 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.036+0000 7f1cc6ffd640 1 -- 172.21.15.12:0/148199610 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721432735 0 0) 0x7f1cc80b8790 con 0x7f1cc815ddc0 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1ccc844640 1 --2- 172.21.15.12:0/148199610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80b8300 0x7f1cc81688d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009e50 tx=0x7f1cc002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2adb7390b03d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cc6ffd640 1 -- 172.21.15.12:0/148199610 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc002c960 con 0x7f1cc80b8300 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cc6ffd640 1 -- 172.21.15.12:0/148199610 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc002cb00 con 0x7f1cc80b8300 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cc6ffd640 1 -- 172.21.15.12:0/148199610 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00343f0 con 0x7f1cc80b8300 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 -- 172.21.15.12:0/148199610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80b8300 msgr2=0x7f1cc81688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/148199610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80b8300 0x7f1cc81688d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009e50 tx=0x7f1cc002be40 comp rx=0 tx=0).stop 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 -- 172.21.15.12:0/148199610 shutdown_connections 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/148199610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80b8300 0x7f1cc81688d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 -- 172.21.15.12:0/148199610 >> 172.21.15.12:0/148199610 conn(0x7f1cc8072720 msgr2=0x7f1cc8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 -- 172.21.15.12:0/148199610 shutdown_connections 2024-03-08T08:29:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.037+0000 7f1cce2a7640 1 -- 172.21.15.12:0/148199610 wait complete. 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.038+0000 7f1cce2a7640 1 Processor -- start 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 -- start start 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 0x7f1cc815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 0x7f1cc81f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80b9980 con 0x7f1cc815ddc0 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80d1d20 con 0x7f1cc8156d60 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cce2a7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80d31c0 con 0x7f1cc80d1110 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57676/0 (socket says 172.21.15.12:57676) 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.039+0000 7f1cc77fe640 1 -- 172.21.15.12:0/3736034034 learned_addr learned my addr 172.21.15.12:0/3736034034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1ccc844640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 0x7f1cc81f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc7fff640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 0x7f1cc815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 -- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 msgr2=0x7f1cc815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 0x7f1cc815a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 -- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 msgr2=0x7f1cc81f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 0x7f1cc81f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 -- 172.21.15.12:0/3736034034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc815fb30 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1ccc844640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 0x7f1cc81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc7fff640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 0x7f1cc815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc77fe640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc00c7b0 tx=0x7f1cbc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc018070 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc0072d0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc815fe10 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0047e0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.040+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc815a9d0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.041+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc808bab0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.041+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cbc01b070 con 0x7f1cc815ddc0 2024-03-08T08:29:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.042+0000 7f1cc4ff9640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 0x7f1cb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.042+0000 7f1cc7fff640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 0x7f1cb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.042+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cbc0895b0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.042+0000 7f1cc7fff640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 0x7f1cb406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8000f60 tx=0x7f1cb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.046+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1cc808bab0 con 0x7f1cc815ddc0 2024-03-08T08:29:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.222+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1cc8156b90 con 0x7f1cb406b500 2024-03-08T08:29:09.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.228+0000 7f1cc4ff9640 1 -- 172.21.15.12:0/3736034034 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237405 (secure 0 0 0) 0x7f1cc8156b90 con 0x7f1cb406b500 2024-03-08T08:29:09.232 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 msgr2=0x7f1cb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 0x7f1cb406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8000f60 tx=0x7f1cb8009040 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 msgr2=0x7f1cc8156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc00c7b0 tx=0x7f1cbc00cc70 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 shutdown_connections 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cb406b500 0x7f1cb406d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cc8156d60 0x7f1cc81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc815ddc0 0x7f1cc8156650 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 --2- 172.21.15.12:0/3736034034 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cc80d1110 0x7f1cc815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 >> 172.21.15.12:0/3736034034 conn(0x7f1cc8072720 msgr2=0x7f1cc815ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 shutdown_connections 2024-03-08T08:29:09.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.231+0000 7f1cce2a7640 1 -- 172.21.15.12:0/3736034034 wait complete. 2024-03-08T08:29:09.243 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-08T08:29:08.993302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239620,"kb_used_data":2452,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735100,"statfs":{"total":773094113280,"available":772848742400,"internally_reserved":0,"allocated":2510848,"data_stored":1576281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.761049"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-08T08:29:04.234876+0000","last_change":"2024-03-08T08:29:04.234876+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.234876+0000","last_undegraded":"2024-03-08T08:29:04.234876+0000","last_fullsized":"2024-03-08T08:29:04.234876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-08T08:29:04.235225+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:04.235225+0000","last_peered":"2024-03-08T08:29:04.235225+0000","last_clean":"2024-03-08T08:29:04.235225+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:04.235225+0000","last_undegraded":"2024-03-08T08:29:04.235225+0000","last_fullsized":"2024-03-08T08:29:04.235225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":644901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542146,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36264,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335576,"statfs":{"total":96636764160,"available":96599629824,"internally_reserved":0,"allocated":757760,"data_stored":639084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:09.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 Processor -- start 2024-03-08T08:29:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 -- start start 2024-03-08T08:29:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35200d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f35200d0f40 0x7f35200d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35200d26f0 0x7f35200d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35200b9030 con 0x7f35200d26f0 2024-03-08T08:29:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35200d13a0 con 0x7f35200d0f40 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.383+0000 7f3525b81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200d1d20 con 0x7f352015ddc0 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35200d26f0 0x7f35200d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35200d26f0 0x7f35200d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57682/0 (socket says 172.21.15.12:57682) 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 -- 172.21.15.12:0/2604146899 learned_addr learned my addr 172.21.15.12:0/2604146899 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351effd640 1 --2- 172.21.15.12:0/2604146899 >> v2:172.21.15.156:3300/0 conn(0x7f35200d0f40 0x7f35200d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351f7fe640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35200d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 -- 172.21.15.12:0/2604146899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 msgr2=0x7f35200d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35200d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 -- 172.21.15.12:0/2604146899 >> v2:172.21.15.156:3300/0 conn(0x7f35200d0f40 msgr2=0x7f35200d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 --2- 172.21.15.12:0/2604146899 >> v2:172.21.15.156:3300/0 conn(0x7f35200d0f40 0x7f35200d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.384+0000 7f351ffff640 1 -- 172.21.15.12:0/2604146899 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352015e1a0 con 0x7f35200d26f0 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f351ffff640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d26f0 0x7f35200d2b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f352008ccc0 tx=0x7f351802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59f7b20fc1e319e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f351e7fc640 1 -- 172.21.15.12:0/2604146899 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351802ca70 con 0x7f35200d26f0 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f351e7fc640 1 -- 172.21.15.12:0/2604146899 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351802cc10 con 0x7f35200d26f0 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f351e7fc640 1 -- 172.21.15.12:0/2604146899 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35180343f0 con 0x7f35200d26f0 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 -- 172.21.15.12:0/2604146899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d26f0 msgr2=0x7f35200d2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d26f0 0x7f35200d2b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f352008ccc0 tx=0x7f351802bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 -- 172.21.15.12:0/2604146899 shutdown_connections 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d26f0 0x7f35200d2b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 --2- 172.21.15.12:0/2604146899 >> v2:172.21.15.156:3300/0 conn(0x7f35200d0f40 0x7f35200d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 --2- 172.21.15.12:0/2604146899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35200d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 -- 172.21.15.12:0/2604146899 >> 172.21.15.12:0/2604146899 conn(0x7f3520072720 msgr2=0x7f3520072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 -- 172.21.15.12:0/2604146899 shutdown_connections 2024-03-08T08:29:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.385+0000 7f3525b81640 1 -- 172.21.15.12:0/2604146899 wait complete. 2024-03-08T08:29:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 Processor -- start 2024-03-08T08:29:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 -- start start 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35200d26f0 0x7f35200b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35201f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b8d50 con 0x7f35200d0f40 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b8a70 con 0x7f35200d26f0 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.387+0000 7f3525b81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b9980 con 0x7f352015ddc0 2024-03-08T08:29:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57692/0 (socket says 172.21.15.12:57692) 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 -- 172.21.15.12:0/2851883674 learned_addr learned my addr 172.21.15.12:0/2851883674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351effd640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35200d26f0 0x7f35200b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351ffff640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35201f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 -- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 msgr2=0x7f35201f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35201f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 -- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35200d26f0 msgr2=0x7f35200b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35200d26f0 0x7f35200b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 -- 172.21.15.12:0/2851883674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200aa870 con 0x7f35200d0f40 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351ffff640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35201f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f351f7fe640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f351000e870 tx=0x7f351000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510017070 con 0x7f35200d0f40 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351000c940 con 0x7f35200d0f40 2024-03-08T08:29:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35200aae30 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000cd30 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.388+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35200ab3f0 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.389+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35201f4590 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.390+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f351001a070 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.390+0000 7f3517fff640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 0x7f350006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.390+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f351008a350 con 0x7f35200d0f40 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.390+0000 7f351effd640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 0x7f350006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.390+0000 7f351effd640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 0x7f350006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3520161470 tx=0x7f3508009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.394+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f35201f4590 con 0x7f35200d0f40 2024-03-08T08:29:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.574+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35200749c0 con 0x7f350006b4d0 2024-03-08T08:29:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.579+0000 7f3517fff640 1 -- 172.21.15.12:0/2851883674 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237405 (secure 0 0 0) 0x7f35200749c0 con 0x7f350006b4d0 2024-03-08T08:29:09.582 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 msgr2=0x7f350006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 0x7f350006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3520161470 tx=0x7f3508009190 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 msgr2=0x7f35201f3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f351000e870 tx=0x7f351000ed30 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 shutdown_connections 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f350006b4d0 0x7f350006d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f352015ddc0 0x7f35201f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35200d26f0 0x7f35200b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 --2- 172.21.15.12:0/2851883674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35200d0f40 0x7f35201f3ff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 >> 172.21.15.12:0/2851883674 conn(0x7f3520072720 msgr2=0x7f35200b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 shutdown_connections 2024-03-08T08:29:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.582+0000 7f3525b81640 1 -- 172.21.15.12:0/2851883674 wait complete. 2024-03-08T08:29:09.594 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-08T08:29:08.993302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239620,"kb_used_data":2452,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735100,"statfs":{"total":773094113280,"available":772848742400,"internally_reserved":0,"allocated":2510848,"data_stored":1576281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.761049"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-08T08:29:04.234876+0000","last_change":"2024-03-08T08:29:04.234876+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.234876+0000","last_undegraded":"2024-03-08T08:29:04.234876+0000","last_fullsized":"2024-03-08T08:29:04.234876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-08T08:29:04.235225+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:04.235225+0000","last_peered":"2024-03-08T08:29:04.235225+0000","last_clean":"2024-03-08T08:29:04.235225+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:04.235225+0000","last_undegraded":"2024-03-08T08:29:04.235225+0000","last_fullsized":"2024-03-08T08:29:04.235225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":644901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542146,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36264,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335576,"statfs":{"total":96636764160,"available":96599629824,"internally_reserved":0,"allocated":757760,"data_stored":639084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:09.597 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.734+0000 7f1e63de2640 1 Processor -- start 2024-03-08T08:29:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.734+0000 7f1e63de2640 1 -- start start 2024-03-08T08:29:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e63de2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 0x7f1e5c0c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e63de2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 0x7f1e5c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e63de2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0cf880 con 0x7f1e5c15aa70 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e63de2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5c15ae50 con 0x7f1e5c0d0d30 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e63de2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0cfb10 con 0x7f1e5c0cf410 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6137d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 0x7f1e5c0c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6237f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 0x7f1e5c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e61b7e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1e5c15aa70 0x7f1e5c0d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47016/0 (socket says 172.21.15.12:47016) 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e61b7e640 1 -- 172.21.15.12:0/176745881 learned_addr learned my addr 172.21.15.12:0/176745881 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e60b7c640 1 -- 172.21.15.12:0/176745881 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201278761 0 0) 0x7f1e5c0cf880 con 0x7f1e5c15aa70 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6237f640 1 -- 172.21.15.12:0/176745881 >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 msgr2=0x7f1e5c0c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6237f640 1 --2- 172.21.15.12:0/176745881 >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 0x7f1e5c0c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6237f640 1 -- 172.21.15.12:0/176745881 >> v1:172.21.15.156:6789/0 conn(0x7f1e5c15aa70 legacy=0x7f1e5c0d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6237f640 1 -- 172.21.15.12:0/176745881 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c0cfdf0 con 0x7f1e5c0cf410 2024-03-08T08:29:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.735+0000 7f1e6137d640 1 --2- 172.21.15.12:0/176745881 >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 0x7f1e5c0c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.736+0000 7f1e6237f640 1 --2- 172.21.15.12:0/176745881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 0x7f1e5c0cb580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097a0 tx=0x7f1e4c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ebf2b2ab694962 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.736+0000 7f1e60b7c640 1 -- 172.21.15.12:0/176745881 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c035020 con 0x7f1e5c0cf410 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.736+0000 7f1e60b7c640 1 -- 172.21.15.12:0/176745881 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4c031830 con 0x7f1e5c0cf410 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.736+0000 7f1e60b7c640 1 -- 172.21.15.12:0/176745881 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c031c00 con 0x7f1e5c0cf410 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 -- 172.21.15.12:0/176745881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 msgr2=0x7f1e5c0cb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 --2- 172.21.15.12:0/176745881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 0x7f1e5c0cb580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097a0 tx=0x7f1e4c02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 -- 172.21.15.12:0/176745881 shutdown_connections 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 --2- 172.21.15.12:0/176745881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c0cf410 0x7f1e5c0cb580 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 --2- 172.21.15.12:0/176745881 >> v2:172.21.15.156:3300/0 conn(0x7f1e5c0d0d30 0x7f1e5c0c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 -- 172.21.15.12:0/176745881 >> 172.21.15.12:0/176745881 conn(0x7f1e5c099a40 msgr2=0x7f1e5c099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 -- 172.21.15.12:0/176745881 shutdown_connections 2024-03-08T08:29:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.737+0000 7f1e63de2640 1 -- 172.21.15.12:0/176745881 wait complete. 2024-03-08T08:29:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.738+0000 7f1e63de2640 1 Processor -- start 2024-03-08T08:29:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.738+0000 7f1e63de2640 1 -- start start 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c0d0d30 0x7f1e5c155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c15aa70 0x7f1e5c0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0cbac0 con 0x7f1e5c0cf410 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0cb060 con 0x7f1e5c0d0d30 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e63de2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0cbc40 con 0x7f1e5c15aa70 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57708/0 (socket says 172.21.15.12:57708) 2024-03-08T08:29:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 -- 172.21.15.12:0/2320385773 learned_addr learned my addr 172.21.15.12:0/2320385773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e6137d640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c0d0d30 0x7f1e5c155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e6237f640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c15aa70 0x7f1e5c0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 -- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c15aa70 msgr2=0x7f1e5c0c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c15aa70 0x7f1e5c0c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 -- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c0d0d30 msgr2=0x7f1e5c155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c0d0d30 0x7f1e5c155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 -- 172.21.15.12:0/2320385773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c155550 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.739+0000 7f1e61b7e640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c7c0 tx=0x7f1e5400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54018070 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5400a640 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c15c160 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400aa30 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c15c720 con 0x7f1e5c0cf410 2024-03-08T08:29:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.740+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c0749c0 con 0x7f1e5c0cf410 2024-03-08T08:29:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.741+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e5401b070 con 0x7f1e5c0cf410 2024-03-08T08:29:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.741+0000 7f1e5a7fc640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 0x7f1e4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.741+0000 7f1e6137d640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 0x7f1e4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.742+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e54089760 con 0x7f1e5c0cf410 2024-03-08T08:29:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.742+0000 7f1e6137d640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 0x7f1e4806d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009560 tx=0x7f1e50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.746+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e54058670 con 0x7f1e5c0cf410 2024-03-08T08:29:09.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.923+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e5c0c71e0 con 0x7f1e4806b500 2024-03-08T08:29:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.928+0000 7f1e5a7fc640 1 -- 172.21.15.12:0/2320385773 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237405 (secure 0 0 0) 0x7f1e5c0c71e0 con 0x7f1e4806b500 2024-03-08T08:29:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:09.934 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:09.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 msgr2=0x7f1e4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 0x7f1e4806d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009560 tx=0x7f1e50008040 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 msgr2=0x7f1e5c0b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c7c0 tx=0x7f1e5400cc80 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 shutdown_connections 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1e4806b500 0x7f1e4806d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e5c15aa70 0x7f1e5c0c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c0d0d30 0x7f1e5c155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 --2- 172.21.15.12:0/2320385773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c0cf410 0x7f1e5c0b1000 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 >> 172.21.15.12:0/2320385773 conn(0x7f1e5c099a40 msgr2=0x7f1e5c1592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 shutdown_connections 2024-03-08T08:29:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:09.931+0000 7f1e63de2640 1 -- 172.21.15.12:0/2320385773 wait complete. 2024-03-08T08:29:09.943 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-08T08:29:08.993302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239620,"kb_used_data":2452,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735100,"statfs":{"total":773094113280,"available":772848742400,"internally_reserved":0,"allocated":2510848,"data_stored":1576281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.761049"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-08T08:29:04.234876+0000","last_change":"2024-03-08T08:29:04.234876+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.234876+0000","last_undegraded":"2024-03-08T08:29:04.234876+0000","last_fullsized":"2024-03-08T08:29:04.234876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-08T08:29:04.235225+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:04.235225+0000","last_peered":"2024-03-08T08:29:04.235225+0000","last_clean":"2024-03-08T08:29:04.235225+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:04.235225+0000","last_undegraded":"2024-03-08T08:29:04.235225+0000","last_fullsized":"2024-03-08T08:29:04.235225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":644901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542146,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36264,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335576,"statfs":{"total":96636764160,"available":96599629824,"internally_reserved":0,"allocated":757760,"data_stored":639084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:09.946 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3f is not active+clean 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3e is not active+clean 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3d is not active+clean 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3c is not active+clean 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3b is not active+clean 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3a is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.39 is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.38 is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.37 is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.36 is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.35 is not active+clean 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.34 is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.33 is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.32 is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.31 is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.30 is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2f is not active+clean 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2e is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2d is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2c is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2b is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2a is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.29 is not active+clean 2024-03-08T08:29:09.950 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.28 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.27 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.26 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.25 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.24 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.23 is not active+clean 2024-03-08T08:29:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.22 is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.21 is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.20 is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1a is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1b is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1b is not active+clean 2024-03-08T08:29:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1a is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.19 is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.18 is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.18 is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.19 is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.17 is not active+clean 2024-03-08T08:29:09.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-08T08:29:04.234876+0000', 'last_change': '2024-03-08T08:29:04.234876+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.234876+0000', 'last_undegraded': '2024-03-08T08:29:04.234876+0000', 'last_fullsized': '2024-03-08T08:29:04.234876+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.16 is not active+clean 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.16 is not active+clean 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.17 is not active+clean 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.15 is not active+clean 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.14 is not active+clean 2024-03-08T08:29:09.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.14 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.15 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.13 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.12 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.12 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.13 is not active+clean 2024-03-08T08:29:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.11 is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.10 is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.10 is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.11 is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.f is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.e is not active+clean 2024-03-08T08:29:09.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.e is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.f is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.d is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.c is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.c is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.d is not active+clean 2024-03-08T08:29:09.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.3 is not active+clean 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2 is not active+clean 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.0 is not active+clean 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1 is not active+clean 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1 is not active+clean 2024-03-08T08:29:09.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.0 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.2 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.4 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.5 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.5 is not active+clean 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.4 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.6 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.7 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.7 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.6 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.8 is not active+clean 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.9 is not active+clean 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.9 is not active+clean 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.8 is not active+clean 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.a is not active+clean 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.b is not active+clean 2024-03-08T08:29:09.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.b is not active+clean 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.a is not active+clean 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1d is not active+clean 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1c is not active+clean 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1c is not active+clean 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1d is not active+clean 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1f is not active+clean 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1e is not active+clean 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1e is not active+clean 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:05.224783+0000', 'last_change': '2024-03-08T08:29:05.224783+0000', 'last_active': '2024-03-08T08:29:05.224783+0000', 'last_peered': '2024-03-08T08:29:05.224783+0000', 'last_clean': '2024-03-08T08:29:05.224783+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:05.224783+0000', 'last_undegraded': '2024-03-08T08:29:05.224783+0000', 'last_fullsized': '2024-03-08T08:29:05.224783+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:05.224783+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1f is not active+clean 2024-03-08T08:29:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-08T08:29:04.219655+0000', 'last_change': '2024-03-08T08:29:04.219655+0000', 'last_active': '2024-03-08T08:29:04.219655+0000', 'last_peered': '2024-03-08T08:29:04.219655+0000', 'last_clean': '2024-03-08T08:29:04.219655+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-08T08:29:04.219655+0000', 'last_undegraded': '2024-03-08T08:29:04.219655+0000', 'last_fullsized': '2024-03-08T08:29:04.219655+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'last_clean_scrub_stamp': '2024-03-08T08:29:04.219655+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-08T08:29:09.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-08T08:29:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.082+0000 7f1540335640 1 Processor -- start 2024-03-08T08:29:10.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.082+0000 7f1540335640 1 -- start start 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f1540335640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1538099c00 0x7f15380c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f1540335640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15380b24c0 0x7f15380ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f1540335640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153809ad80 con 0x7f153809a880 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f1540335640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15380cf570 con 0x7f15380b24c0 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f1540335640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15380cf850 con 0x7f1538099c00 2024-03-08T08:29:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f153e8d2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15380b24c0 0x7f15380ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f153e8d2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15380b24c0 0x7f15380ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57714/0 (socket says 172.21.15.12:57714) 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.083+0000 7f153e8d2640 1 -- 172.21.15.12:0/974252945 learned_addr learned my addr 172.21.15.12:0/974252945 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e0d1640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1538099c00 0x7f15380c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e8d2640 1 -- 172.21.15.12:0/974252945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1538099c00 msgr2=0x7f15380c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e8d2640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1538099c00 0x7f15380c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e8d2640 1 -- 172.21.15.12:0/974252945 >> v1:172.21.15.156:6789/0 conn(0x7f153809a880 legacy=0x7f1538099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e8d2640 1 -- 172.21.15.12:0/974252945 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15380cfb30 con 0x7f15380b24c0 2024-03-08T08:29:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153e8d2640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15380b24c0 0x7f15380ccad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1530009870 tx=0x7f153002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77536fdd305e712c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.084+0000 7f153d0cf640 1 -- 172.21.15.12:0/974252945 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530034070 con 0x7f15380b24c0 2024-03-08T08:29:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f153d0cf640 1 -- 172.21.15.12:0/974252945 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1530030840 con 0x7f15380b24c0 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f153d0cf640 1 -- 172.21.15.12:0/974252945 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530030c10 con 0x7f15380b24c0 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 -- 172.21.15.12:0/974252945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15380b24c0 msgr2=0x7f15380ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15380b24c0 0x7f15380ccad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1530009870 tx=0x7f153002b980 comp rx=0 tx=0).stop 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 -- 172.21.15.12:0/974252945 shutdown_connections 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15380b24c0 0x7f15380ccad0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 --2- 172.21.15.12:0/974252945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1538099c00 0x7f15380c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 -- 172.21.15.12:0/974252945 >> 172.21.15.12:0/974252945 conn(0x7f15380b3490 msgr2=0x7f15380b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 -- 172.21.15.12:0/974252945 shutdown_connections 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.085+0000 7f1540335640 1 -- 172.21.15.12:0/974252945 wait complete. 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.086+0000 7f1540335640 1 Processor -- start 2024-03-08T08:29:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.086+0000 7f1540335640 1 -- start start 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 0x7f153815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 0x7f153815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15380ce240 con 0x7f1538099c00 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15380ce0c0 con 0x7f15380b24c0 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f1540335640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15380cd010 con 0x7f153809a880 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57716/0 (socket says 172.21.15.12:57716) 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 -- 172.21.15.12:0/2398626997 learned_addr learned my addr 172.21.15.12:0/2398626997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e8d2640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 0x7f153815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153d8d0640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 0x7f153815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 -- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 msgr2=0x7f153815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 0x7f153815c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 -- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 msgr2=0x7f153815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 0x7f153815c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e0d1640 1 -- 172.21.15.12:0/2398626997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15381570b0 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153e8d2640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 0x7f153815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.087+0000 7f153d8d0640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 0x7f153815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f153e0d1640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1528007f20 tx=0x7f152800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528014620 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1528019070 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1538157390 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528014a00 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.088+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1538157950 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.089+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15380749c0 con 0x7f1538099c00 2024-03-08T08:29:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.089+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1528014ba0 con 0x7f1538099c00 2024-03-08T08:29:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.089+0000 7f1536ffd640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 0x7f152406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.090+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f152808ab60 con 0x7f1538099c00 2024-03-08T08:29:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.090+0000 7f153d8d0640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 0x7f152406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.090+0000 7f153d8d0640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 0x7f152406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15381f5a40 tx=0x7f152c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.094+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f15280599d0 con 0x7f1538099c00 2024-03-08T08:29:10.287 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-08T08:29:10.286+0000 7f0963ea2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-08T08:29:10.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.343+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f15380b42f0 con 0x7f1538099c00 2024-03-08T08:29:10.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.343+0000 7f1536ffd640 1 -- 172.21.15.12:0/2398626997 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1107 (secure 0 0 0) 0x7f15280592d0 con 0x7f1538099c00 2024-03-08T08:29:10.344 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:10.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":21,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1709886539,"num_in_osds":8,"osd_in_since":1709886530,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":95},{"state_name":"active+clean","count":1},{"state_name":"creating+peering","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":245370880,"bytes_avail":772848742400,"bytes_total":773094113280,"unknown_pgs_ratio":0.97938144207000732,"inactive_pgs_ratio":0.010309278033673763},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"a","status":"up:active","gid":4254}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-08T08:28:48.657483+0000","services":{}},"progress_events":{}} 2024-03-08T08:29:10.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 msgr2=0x7f152406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 0x7f152406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15381f5a40 tx=0x7f152c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:10.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 msgr2=0x7f1538156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1528007f20 tx=0x7f152800e570 comp rx=0 tx=0).stop 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 shutdown_connections 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f152406b4d0 0x7f152406d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15380b24c0 0x7f153815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f153809a880 0x7f153815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 --2- 172.21.15.12:0/2398626997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1538099c00 0x7f1538156a20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 >> 172.21.15.12:0/2398626997 conn(0x7f15380b3490 msgr2=0x7f15380c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 shutdown_connections 2024-03-08T08:29:10.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.345+0000 7f1540335640 1 -- 172.21.15.12:0/2398626997 wait complete. 2024-03-08T08:29:10.356 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-08T08:29:10.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:10.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.456+0000 7f1635348640 1 Processor -- start 2024-03-08T08:29:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.456+0000 7f1635348640 1 -- start start 2024-03-08T08:29:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f1635348640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f16300c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f1635348640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163009ad80 con 0x7f163009a880 2024-03-08T08:29:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f1635348640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16300cf570 con 0x7f16300b24c0 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f1635348640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300cf850 con 0x7f1630099c00 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f162e7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f163009a880 0x7f1630099fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52760/0 (socket says 172.21.15.12:52760) 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f162e7fc640 1 -- 172.21.15.12:0/1663802608 learned_addr learned my addr 172.21.15.12:0/1663802608 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f162effd640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f16300c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.457+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104213090 0 0) 0x7f163009ad80 con 0x7f163009a880 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161c003520 con 0x7f163009a880 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011749062 0 0) 0x7f16300cf570 con 0x7f16300b24c0 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f163009ad80 con 0x7f16300b24c0 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 238999744 0 0) 0x7f161c003520 con 0x7f163009a880 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 msgr2=0x7f16300c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f16300c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 >> v1:172.21.15.156:6789/0 conn(0x7f16300b24c0 legacy=0x7f16300ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300cfb30 con 0x7f163009a880 2024-03-08T08:29:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f1618001670 con 0x7f163009a880 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 >> v1:172.21.15.12:6789/0 conn(0x7f163009a880 legacy=0x7f1630099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f161c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c004120 0x7f161c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300cf850 con 0x7f161c004d40 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c00bb00 con 0x7f161c004120 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c00bd00 con 0x7f1630099c00 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162effd640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f161c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162e7fc640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c004120 0x7f161c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 -- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 msgr2=0x7f161c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1630099c00 0x7f161c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 -- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c004120 msgr2=0x7f161c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:10.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c004120 0x7f161c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162f7fe640 1 -- 172.21.15.12:0/1663802608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c003520 con 0x7f161c004d40 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.458+0000 7f162e7fc640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c004120 0x7f161c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.459+0000 7f162f7fe640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16280018d0 tx=0x7f161c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd3a6d963914fa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.459+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628037070 con 0x7f161c004d40 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.459+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16280052d0 con 0x7f161c004d40 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.459+0000 7f162dffb640 1 -- 172.21.15.12:0/1663802608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16280014f0 con 0x7f161c004d40 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 -- 172.21.15.12:0/1663802608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c004d40 msgr2=0x7f161c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 --2- 172.21.15.12:0/1663802608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16280018d0 tx=0x7f161c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f162f7fe640 1 -- 172.21.15.12:0/1663802608 reap_dead start 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 -- 172.21.15.12:0/1663802608 shutdown_connections 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 -- 172.21.15.12:0/1663802608 >> 172.21.15.12:0/1663802608 conn(0x7f16300b3490 msgr2=0x7f16300b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 -- 172.21.15.12:0/1663802608 shutdown_connections 2024-03-08T08:29:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.460+0000 7f1635348640 1 -- 172.21.15.12:0/1663802608 wait complete. 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.461+0000 7f1635348640 1 Processor -- start 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.461+0000 7f1635348640 1 -- start start 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16300c48b0 0x7f16301f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f163015b500 0x7f163006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300ce240 con 0x7f163015de50 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300ce0c0 con 0x7f16300c48b0 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c54f0 con 0x7f163015b500 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57732/0 (socket says 172.21.15.12:57732) 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 -- 172.21.15.12:0/2380060554 learned_addr learned my addr 172.21.15.12:0/2380060554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162effd640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16300c48b0 0x7f16301f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162e7fc640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f163015b500 0x7f163006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 -- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f163015b500 msgr2=0x7f163006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f163015b500 0x7f163006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 -- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16300c48b0 msgr2=0x7f16301f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16300c48b0 0x7f16301f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 -- 172.21.15.12:0/2380060554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163015b9c0 con 0x7f163015de50 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f162f7fe640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f16280018d0 tx=0x7f16280012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628037070 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628038040 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.462+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163015bca0 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.463+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16280014f0 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.463+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163015c260 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.464+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16300749c0 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.464+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16280052d0 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.464+0000 7f16277fe640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 0x7f161006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.464+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16280ab080 con 0x7f163015de50 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.464+0000 7f162effd640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 0x7f161006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.465+0000 7f162effd640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 0x7f161006d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f163015f120 tx=0x7f162000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.469+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1628079940 con 0x7f163015de50 2024-03-08T08:29:10.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.649+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16301f8eb0 con 0x7f161006b500 2024-03-08T08:29:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.654+0000 7f16277fe640 1 -- 172.21.15.12:0/2380060554 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237405 (secure 0 0 0) 0x7f1620009e20 con 0x7f161006b500 2024-03-08T08:29:10.658 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 msgr2=0x7f161006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 0x7f161006d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f163015f120 tx=0x7f162000b040 comp rx=0 tx=0).stop 2024-03-08T08:29:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 msgr2=0x7f163015e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f16280018d0 tx=0x7f16280012d0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 shutdown_connections 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f161006b500 0x7f161006d9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f163015de50 0x7f163015e2e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f163015b500 0x7f163006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 --2- 172.21.15.12:0/2380060554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16300c48b0 0x7f16301f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 >> 172.21.15.12:0/2380060554 conn(0x7f16300b3490 msgr2=0x7f16300c6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.657+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 shutdown_connections 2024-03-08T08:29:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:10.658+0000 7f1635348640 1 -- 172.21.15.12:0/2380060554 wait complete. 2024-03-08T08:29:10.669 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-08T08:29:08.993302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239620,"kb_used_data":2452,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735100,"statfs":{"total":773094113280,"available":772848742400,"internally_reserved":0,"allocated":2510848,"data_stored":1576281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.761049"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-08T08:29:04.234876+0000","last_change":"2024-03-08T08:29:04.234876+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.234876+0000","last_undegraded":"2024-03-08T08:29:04.234876+0000","last_fullsized":"2024-03-08T08:29:04.234876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-08T08:29:04.235225+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:04.235225+0000","last_peered":"2024-03-08T08:29:04.235225+0000","last_clean":"2024-03-08T08:29:04.235225+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:04.235225+0000","last_undegraded":"2024-03-08T08:29:04.235225+0000","last_fullsized":"2024-03-08T08:29:04.235225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:05.224783+0000","last_change":"2024-03-08T08:29:05.224783+0000","last_active":"2024-03-08T08:29:05.224783+0000","last_peered":"2024-03-08T08:29:05.224783+0000","last_clean":"2024-03-08T08:29:05.224783+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:05.224783+0000","last_undegraded":"2024-03-08T08:29:05.224783+0000","last_fullsized":"2024-03-08T08:29:05.224783+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:29:04.219655+0000","last_change":"2024-03-08T08:29:04.219655+0000","last_active":"2024-03-08T08:29:04.219655+0000","last_peered":"2024-03-08T08:29:04.219655+0000","last_clean":"2024-03-08T08:29:04.219655+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:29:04.219655+0000","last_undegraded":"2024-03-08T08:29:04.219655+0000","last_fullsized":"2024-03-08T08:29:04.219655+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":644901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542146,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36264,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335576,"statfs":{"total":96636764160,"available":96599629824,"internally_reserved":0,"allocated":757760,"data_stored":639084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":48716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:13.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.784+0000 7f83c2a91640 1 Processor -- start 2024-03-08T08:29:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.784+0000 7f83c2a91640 1 -- start start 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c2a91640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c2a91640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 0x7f83bc0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c2a91640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc15e1a0 con 0x7f83bc0d26f0 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c2a91640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc0b9030 con 0x7f83bc0d0f40 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c2a91640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d13a0 con 0x7f83bc15ddc0 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c102e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f83bc0d26f0 0x7f83bc0d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52764/0 (socket says 172.21.15.12:52764) 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c102e640 1 -- 172.21.15.12:0/2443597413 learned_addr learned my addr 172.21.15.12:0/2443597413 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83bbfff640 1 --2- 172.21.15.12:0/2443597413 >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 0x7f83bc0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.785+0000 7f83c082d640 1 --2- 172.21.15.12:0/2443597413 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816252602 0 0) 0x7f83bc15e1a0 con 0x7f83bc0d26f0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a4003520 con 0x7f83bc0d26f0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1672452819 0 0) 0x7f83a4003520 con 0x7f83bc0d26f0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83c082d640 1 -- 172.21.15.12:0/2443597413 >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 msgr2=0x7f83bc0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83c082d640 1 --2- 172.21.15.12:0/2443597413 >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 0x7f83bc0d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83c082d640 1 -- 172.21.15.12:0/2443597413 >> v1:172.21.15.12:6789/0 conn(0x7f83bc0d26f0 legacy=0x7f83bc0d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83c082d640 1 -- 172.21.15.12:0/2443597413 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0b8790 con 0x7f83bc15ddc0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.786+0000 7f83bbfff640 1 --2- 172.21.15.12:0/2443597413 >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 0x7f83bc0d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c082d640 1 --2- 172.21.15.12:0/2443597413 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc0d0a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009d90 tx=0x7f83b402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45dc77b802ff1172 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4039070 con 0x7f83bc15ddc0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b402c8a0 con 0x7f83bc15ddc0 2024-03-08T08:29:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83bb7fe640 1 -- 172.21.15.12:0/2443597413 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac001670 con 0x7f83bc15ddc0 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 -- 172.21.15.12:0/2443597413 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 msgr2=0x7f83bc0d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 --2- 172.21.15.12:0/2443597413 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc0d0a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009d90 tx=0x7f83b402bd70 comp rx=0 tx=0).stop 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 -- 172.21.15.12:0/2443597413 shutdown_connections 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 --2- 172.21.15.12:0/2443597413 >> v2:172.21.15.156:3300/0 conn(0x7f83bc0d0f40 0x7f83bc0d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 --2- 172.21.15.12:0/2443597413 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc0d0a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.787+0000 7f83c2a91640 1 -- 172.21.15.12:0/2443597413 >> 172.21.15.12:0/2443597413 conn(0x7f83bc072720 msgr2=0x7f83bc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.788+0000 7f83c2a91640 1 -- 172.21.15.12:0/2443597413 shutdown_connections 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.788+0000 7f83c2a91640 1 -- 172.21.15.12:0/2443597413 wait complete. 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.789+0000 7f83c2a91640 1 Processor -- start 2024-03-08T08:29:13.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.789+0000 7f83c2a91640 1 -- start start 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83bc0d26f0 0x7f83bc1f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc160650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b8d50 con 0x7f83bc0d0f40 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d1d20 con 0x7f83bc0d26f0 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b9980 con 0x7f83bc15ddc0 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57740/0 (socket says 172.21.15.12:57740) 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 -- 172.21.15.12:0/495499306 learned_addr learned my addr 172.21.15.12:0/495499306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83bbfff640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83bc0d26f0 0x7f83bc1f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 -- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 msgr2=0x7f83bc160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 -- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83bc0d26f0 msgr2=0x7f83bc1f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83bc0d26f0 0x7f83bc1f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 -- 172.21.15.12:0/495499306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc15f5b0 con 0x7f83bc0d0f40 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c082d640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83b4037040 tx=0x7f83b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4039050 con 0x7f83bc0d0f40 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4045070 con 0x7f83bc0d0f40 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc15f890 con 0x7f83bc0d0f40 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.790+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4009450 con 0x7f83bc0d0f40 2024-03-08T08:29:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.791+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc15fe50 con 0x7f83bc0d0f40 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.791+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc0749c0 con 0x7f83bc0d0f40 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.792+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83b4040690 con 0x7f83bc0d0f40 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.792+0000 7f83b97fa640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.792+0000 7f83bbfff640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.792+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83b40acbc0 con 0x7f83bc0d0f40 2024-03-08T08:29:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.793+0000 7f83bbfff640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 0x7f83a806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009560 tx=0x7f83b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.797+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83b407bb60 con 0x7f83bc0d0f40 2024-03-08T08:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.972+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83bc0d0ba0 con 0x7f83a806b4d0 2024-03-08T08:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.978+0000 7f83b97fa640 1 -- 172.21.15.12:0/495499306 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251922 (secure 0 0 0) 0x7f83bc0d0ba0 con 0x7f83a806b4d0 2024-03-08T08:29:13.982 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 msgr2=0x7f83a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 0x7f83a806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009560 tx=0x7f83b0008040 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 msgr2=0x7f83bc0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83b4037040 tx=0x7f83b4004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 shutdown_connections 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83bc15ddc0 0x7f83bc160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83bc0d26f0 0x7f83bc1f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 --2- 172.21.15.12:0/495499306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83bc0d0f40 0x7f83bc0acdd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 >> 172.21.15.12:0/495499306 conn(0x7f83bc072720 msgr2=0x7f83bc0b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 shutdown_connections 2024-03-08T08:29:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:13.981+0000 7f83c2a91640 1 -- 172.21.15.12:0/495499306 wait complete. 2024-03-08T08:29:13.993 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-08T08:29:12.994264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239856,"kb_used_data":2864,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734864,"statfs":{"total":773094113280,"available":772848500736,"internally_reserved":0,"allocated":2932736,"data_stored":1764020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.762012"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562754+0000","last_change":"2024-03-08T08:29:06.251346+0000","last_active":"2024-03-08T08:29:07.562754+0000","last_peered":"2024-03-08T08:29:07.562754+0000","last_clean":"2024-03-08T08:29:07.562754+0000","last_became_active":"2024-03-08T08:29:06.250986+0000","last_became_peered":"2024-03-08T08:29:06.250986+0000","last_unstale":"2024-03-08T08:29:07.562754+0000","last_undegraded":"2024-03-08T08:29:07.562754+0000","last_fullsized":"2024-03-08T08:29:07.562754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:31:13.095757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905464+0000","last_change":"2024-03-08T08:29:06.247932+0000","last_active":"2024-03-08T08:29:07.905464+0000","last_peered":"2024-03-08T08:29:07.905464+0000","last_clean":"2024-03-08T08:29:07.905464+0000","last_became_active":"2024-03-08T08:29:06.247623+0000","last_became_peered":"2024-03-08T08:29:06.247623+0000","last_unstale":"2024-03-08T08:29:07.905464+0000","last_undegraded":"2024-03-08T08:29:07.905464+0000","last_fullsized":"2024-03-08T08:29:07.905464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:48:05.979625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905335+0000","last_change":"2024-03-08T08:29:06.244061+0000","last_active":"2024-03-08T08:29:07.905335+0000","last_peered":"2024-03-08T08:29:07.905335+0000","last_clean":"2024-03-08T08:29:07.905335+0000","last_became_active":"2024-03-08T08:29:06.243801+0000","last_became_peered":"2024-03-08T08:29:06.243801+0000","last_unstale":"2024-03-08T08:29:07.905335+0000","last_undegraded":"2024-03-08T08:29:07.905335+0000","last_fullsized":"2024-03-08T08:29:07.905335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:49:40.562208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.253434+0000","last_change":"2024-03-08T08:29:06.250895+0000","last_active":"2024-03-08T08:29:06.253434+0000","last_peered":"2024-03-08T08:29:06.253434+0000","last_clean":"2024-03-08T08:29:06.253434+0000","last_became_active":"2024-03-08T08:29:06.250560+0000","last_became_peered":"2024-03-08T08:29:06.250560+0000","last_unstale":"2024-03-08T08:29:06.253434+0000","last_undegraded":"2024-03-08T08:29:06.253434+0000","last_fullsized":"2024-03-08T08:29:06.253434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:45.498239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562498+0000","last_change":"2024-03-08T08:29:06.249635+0000","last_active":"2024-03-08T08:29:07.562498+0000","last_peered":"2024-03-08T08:29:07.562498+0000","last_clean":"2024-03-08T08:29:07.562498+0000","last_became_active":"2024-03-08T08:29:06.249356+0000","last_became_peered":"2024-03-08T08:29:06.249356+0000","last_unstale":"2024-03-08T08:29:07.562498+0000","last_undegraded":"2024-03-08T08:29:07.562498+0000","last_fullsized":"2024-03-08T08:29:07.562498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:47:43.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.250964+0000","last_change":"2024-03-08T08:29:06.249994+0000","last_active":"2024-03-08T08:29:06.250964+0000","last_peered":"2024-03-08T08:29:06.250964+0000","last_clean":"2024-03-08T08:29:06.250964+0000","last_became_active":"2024-03-08T08:29:06.249682+0000","last_became_peered":"2024-03-08T08:29:06.249682+0000","last_unstale":"2024-03-08T08:29:06.250964+0000","last_undegraded":"2024-03-08T08:29:06.250964+0000","last_fullsized":"2024-03-08T08:29:06.250964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:29:46.094628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562203+0000","last_change":"2024-03-08T08:29:06.237963+0000","last_active":"2024-03-08T08:29:07.562203+0000","last_peered":"2024-03-08T08:29:07.562203+0000","last_clean":"2024-03-08T08:29:07.562203+0000","last_became_active":"2024-03-08T08:29:06.237654+0000","last_became_peered":"2024-03-08T08:29:06.237654+0000","last_unstale":"2024-03-08T08:29:07.562203+0000","last_undegraded":"2024-03-08T08:29:07.562203+0000","last_fullsized":"2024-03-08T08:29:07.562203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:16:24.171714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905138+0000","last_change":"2024-03-08T08:29:06.258895+0000","last_active":"2024-03-08T08:29:07.905138+0000","last_peered":"2024-03-08T08:29:07.905138+0000","last_clean":"2024-03-08T08:29:07.905138+0000","last_became_active":"2024-03-08T08:29:06.258630+0000","last_became_peered":"2024-03-08T08:29:06.258630+0000","last_unstale":"2024-03-08T08:29:07.905138+0000","last_undegraded":"2024-03-08T08:29:07.905138+0000","last_fullsized":"2024-03-08T08:29:07.905138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:02:10.390227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236928+0000","last_change":"2024-03-08T08:29:06.260239+0000","last_active":"2024-03-08T08:29:07.236928+0000","last_peered":"2024-03-08T08:29:07.236928+0000","last_clean":"2024-03-08T08:29:07.236928+0000","last_became_active":"2024-03-08T08:29:06.259944+0000","last_became_peered":"2024-03-08T08:29:06.259944+0000","last_unstale":"2024-03-08T08:29:07.236928+0000","last_undegraded":"2024-03-08T08:29:07.236928+0000","last_fullsized":"2024-03-08T08:29:07.236928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:18:09.551413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562557+0000","last_change":"2024-03-08T08:29:06.252950+0000","last_active":"2024-03-08T08:29:07.562557+0000","last_peered":"2024-03-08T08:29:07.562557+0000","last_clean":"2024-03-08T08:29:07.562557+0000","last_became_active":"2024-03-08T08:29:06.252480+0000","last_became_peered":"2024-03-08T08:29:06.252480+0000","last_unstale":"2024-03-08T08:29:07.562557+0000","last_undegraded":"2024-03-08T08:29:07.562557+0000","last_fullsized":"2024-03-08T08:29:07.562557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:40:11.278068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562751+0000","last_change":"2024-03-08T08:29:06.248718+0000","last_active":"2024-03-08T08:29:07.562751+0000","last_peered":"2024-03-08T08:29:07.562751+0000","last_clean":"2024-03-08T08:29:07.562751+0000","last_became_active":"2024-03-08T08:29:06.248400+0000","last_became_peered":"2024-03-08T08:29:06.248400+0000","last_unstale":"2024-03-08T08:29:07.562751+0000","last_undegraded":"2024-03-08T08:29:07.562751+0000","last_fullsized":"2024-03-08T08:29:07.562751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:02:07.589132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562439+0000","last_change":"2024-03-08T08:29:06.247528+0000","last_active":"2024-03-08T08:29:07.562439+0000","last_peered":"2024-03-08T08:29:07.562439+0000","last_clean":"2024-03-08T08:29:07.562439+0000","last_became_active":"2024-03-08T08:29:06.246926+0000","last_became_peered":"2024-03-08T08:29:06.246926+0000","last_unstale":"2024-03-08T08:29:07.562439+0000","last_undegraded":"2024-03-08T08:29:07.562439+0000","last_fullsized":"2024-03-08T08:29:07.562439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:42.233464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409364+0000","last_change":"2024-03-08T08:29:06.261811+0000","last_active":"2024-03-08T08:29:07.409364+0000","last_peered":"2024-03-08T08:29:07.409364+0000","last_clean":"2024-03-08T08:29:07.409364+0000","last_became_active":"2024-03-08T08:29:06.261568+0000","last_became_peered":"2024-03-08T08:29:06.261568+0000","last_unstale":"2024-03-08T08:29:07.409364+0000","last_undegraded":"2024-03-08T08:29:07.409364+0000","last_fullsized":"2024-03-08T08:29:07.409364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409154+0000","last_change":"2024-03-08T08:29:06.245702+0000","last_active":"2024-03-08T08:29:07.409154+0000","last_peered":"2024-03-08T08:29:07.409154+0000","last_clean":"2024-03-08T08:29:07.409154+0000","last_became_active":"2024-03-08T08:29:06.245425+0000","last_became_peered":"2024-03-08T08:29:06.245425+0000","last_unstale":"2024-03-08T08:29:07.409154+0000","last_undegraded":"2024-03-08T08:29:07.409154+0000","last_fullsized":"2024-03-08T08:29:07.409154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:01.699386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905113+0000","last_change":"2024-03-08T08:29:06.250908+0000","last_active":"2024-03-08T08:29:07.905113+0000","last_peered":"2024-03-08T08:29:07.905113+0000","last_clean":"2024-03-08T08:29:07.905113+0000","last_became_active":"2024-03-08T08:29:06.250627+0000","last_became_peered":"2024-03-08T08:29:06.250627+0000","last_unstale":"2024-03-08T08:29:07.905113+0000","last_undegraded":"2024-03-08T08:29:07.905113+0000","last_fullsized":"2024-03-08T08:29:07.905113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:26:00.214895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.263972+0000","last_change":"2024-03-08T08:29:06.257106+0000","last_active":"2024-03-08T08:29:06.263972+0000","last_peered":"2024-03-08T08:29:06.263972+0000","last_clean":"2024-03-08T08:29:06.263972+0000","last_became_active":"2024-03-08T08:29:06.256817+0000","last_became_peered":"2024-03-08T08:29:06.256817+0000","last_unstale":"2024-03-08T08:29:06.263972+0000","last_undegraded":"2024-03-08T08:29:06.263972+0000","last_fullsized":"2024-03-08T08:29:06.263972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:14:53.496011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264929+0000","last_change":"2024-03-08T08:29:06.262604+0000","last_active":"2024-03-08T08:29:06.264929+0000","last_peered":"2024-03-08T08:29:06.264929+0000","last_clean":"2024-03-08T08:29:06.264929+0000","last_became_active":"2024-03-08T08:29:06.262222+0000","last_became_peered":"2024-03-08T08:29:06.262222+0000","last_unstale":"2024-03-08T08:29:06.264929+0000","last_undegraded":"2024-03-08T08:29:06.264929+0000","last_fullsized":"2024-03-08T08:29:06.264929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:23:31.113352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235999+0000","last_change":"2024-03-08T08:29:06.256729+0000","last_active":"2024-03-08T08:29:07.235999+0000","last_peered":"2024-03-08T08:29:07.235999+0000","last_clean":"2024-03-08T08:29:07.235999+0000","last_became_active":"2024-03-08T08:29:06.256220+0000","last_became_peered":"2024-03-08T08:29:06.256220+0000","last_unstale":"2024-03-08T08:29:07.235999+0000","last_undegraded":"2024-03-08T08:29:07.235999+0000","last_fullsized":"2024-03-08T08:29:07.235999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:16.445063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.256979+0000","last_change":"2024-03-08T08:29:06.251913+0000","last_active":"2024-03-08T08:29:06.256979+0000","last_peered":"2024-03-08T08:29:06.256979+0000","last_clean":"2024-03-08T08:29:06.256979+0000","last_became_active":"2024-03-08T08:29:06.251609+0000","last_became_peered":"2024-03-08T08:29:06.251609+0000","last_unstale":"2024-03-08T08:29:06.256979+0000","last_undegraded":"2024-03-08T08:29:06.256979+0000","last_fullsized":"2024-03-08T08:29:06.256979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562581+0000","last_change":"2024-03-08T08:29:06.247738+0000","last_active":"2024-03-08T08:29:07.562581+0000","last_peered":"2024-03-08T08:29:07.562581+0000","last_clean":"2024-03-08T08:29:07.562581+0000","last_became_active":"2024-03-08T08:29:06.247254+0000","last_became_peered":"2024-03-08T08:29:06.247254+0000","last_unstale":"2024-03-08T08:29:07.562581+0000","last_undegraded":"2024-03-08T08:29:07.562581+0000","last_fullsized":"2024-03-08T08:29:07.562581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:01:26.719267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235275+0000","last_change":"2024-03-08T08:29:06.251015+0000","last_active":"2024-03-08T08:29:07.235275+0000","last_peered":"2024-03-08T08:29:07.235275+0000","last_clean":"2024-03-08T08:29:07.235275+0000","last_became_active":"2024-03-08T08:29:06.250716+0000","last_became_peered":"2024-03-08T08:29:06.250716+0000","last_unstale":"2024-03-08T08:29:07.235275+0000","last_undegraded":"2024-03-08T08:29:07.235275+0000","last_fullsized":"2024-03-08T08:29:07.235275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236632+0000","last_change":"2024-03-08T08:29:06.259314+0000","last_active":"2024-03-08T08:29:07.236632+0000","last_peered":"2024-03-08T08:29:07.236632+0000","last_clean":"2024-03-08T08:29:07.236632+0000","last_became_active":"2024-03-08T08:29:06.258967+0000","last_became_peered":"2024-03-08T08:29:06.258967+0000","last_unstale":"2024-03-08T08:29:07.236632+0000","last_undegraded":"2024-03-08T08:29:07.236632+0000","last_fullsized":"2024-03-08T08:29:07.236632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:04:34.787442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905406+0000","last_change":"2024-03-08T08:29:06.251496+0000","last_active":"2024-03-08T08:29:07.905406+0000","last_peered":"2024-03-08T08:29:07.905406+0000","last_clean":"2024-03-08T08:29:07.905406+0000","last_became_active":"2024-03-08T08:29:06.251220+0000","last_became_peered":"2024-03-08T08:29:06.251220+0000","last_unstale":"2024-03-08T08:29:07.905406+0000","last_undegraded":"2024-03-08T08:29:07.905406+0000","last_fullsized":"2024-03-08T08:29:07.905406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:31:41.063261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.262007+0000","last_change":"2024-03-08T08:29:06.256404+0000","last_active":"2024-03-08T08:29:06.262007+0000","last_peered":"2024-03-08T08:29:06.262007+0000","last_clean":"2024-03-08T08:29:06.262007+0000","last_became_active":"2024-03-08T08:29:06.256087+0000","last_became_peered":"2024-03-08T08:29:06.256087+0000","last_unstale":"2024-03-08T08:29:06.262007+0000","last_undegraded":"2024-03-08T08:29:06.262007+0000","last_fullsized":"2024-03-08T08:29:06.262007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:18:25.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905562+0000","last_change":"2024-03-08T08:29:06.255787+0000","last_active":"2024-03-08T08:29:07.905562+0000","last_peered":"2024-03-08T08:29:07.905562+0000","last_clean":"2024-03-08T08:29:07.905562+0000","last_became_active":"2024-03-08T08:29:06.255548+0000","last_became_peered":"2024-03-08T08:29:06.255548+0000","last_unstale":"2024-03-08T08:29:07.905562+0000","last_undegraded":"2024-03-08T08:29:07.905562+0000","last_fullsized":"2024-03-08T08:29:07.905562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:28:26.547413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.261082+0000","last_change":"2024-03-08T08:29:06.259612+0000","last_active":"2024-03-08T08:29:06.261082+0000","last_peered":"2024-03-08T08:29:06.261082+0000","last_clean":"2024-03-08T08:29:06.261082+0000","last_became_active":"2024-03-08T08:29:06.259198+0000","last_became_peered":"2024-03-08T08:29:06.259198+0000","last_unstale":"2024-03-08T08:29:06.261082+0000","last_undegraded":"2024-03-08T08:29:06.261082+0000","last_fullsized":"2024-03-08T08:29:06.261082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:05.720825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409477+0000","last_change":"2024-03-08T08:29:06.245080+0000","last_active":"2024-03-08T08:29:07.409477+0000","last_peered":"2024-03-08T08:29:07.409477+0000","last_clean":"2024-03-08T08:29:07.409477+0000","last_became_active":"2024-03-08T08:29:06.244821+0000","last_became_peered":"2024-03-08T08:29:06.244821+0000","last_unstale":"2024-03-08T08:29:07.409477+0000","last_undegraded":"2024-03-08T08:29:07.409477+0000","last_fullsized":"2024-03-08T08:29:07.409477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409429+0000","last_change":"2024-03-08T08:29:06.259724+0000","last_active":"2024-03-08T08:29:07.409429+0000","last_peered":"2024-03-08T08:29:07.409429+0000","last_clean":"2024-03-08T08:29:07.409429+0000","last_became_active":"2024-03-08T08:29:06.259415+0000","last_became_peered":"2024-03-08T08:29:06.259415+0000","last_unstale":"2024-03-08T08:29:07.409429+0000","last_undegraded":"2024-03-08T08:29:07.409429+0000","last_fullsized":"2024-03-08T08:29:07.409429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905249+0000","last_change":"2024-03-08T08:29:06.248138+0000","last_active":"2024-03-08T08:29:07.905249+0000","last_peered":"2024-03-08T08:29:07.905249+0000","last_clean":"2024-03-08T08:29:07.905249+0000","last_became_active":"2024-03-08T08:29:06.247330+0000","last_became_peered":"2024-03-08T08:29:06.247330+0000","last_unstale":"2024-03-08T08:29:07.905249+0000","last_undegraded":"2024-03-08T08:29:07.905249+0000","last_fullsized":"2024-03-08T08:29:07.905249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:40:15.729074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.256829+0000","last_change":"2024-03-08T08:29:06.254998+0000","last_active":"2024-03-08T08:29:06.256829+0000","last_peered":"2024-03-08T08:29:06.256829+0000","last_clean":"2024-03-08T08:29:06.256829+0000","last_became_active":"2024-03-08T08:29:06.254596+0000","last_became_peered":"2024-03-08T08:29:06.254596+0000","last_unstale":"2024-03-08T08:29:06.256829+0000","last_undegraded":"2024-03-08T08:29:06.256829+0000","last_fullsized":"2024-03-08T08:29:06.256829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235098+0000","last_change":"2024-03-08T08:29:06.244995+0000","last_active":"2024-03-08T08:29:07.235098+0000","last_peered":"2024-03-08T08:29:07.235098+0000","last_clean":"2024-03-08T08:29:07.235098+0000","last_became_active":"2024-03-08T08:29:06.244712+0000","last_became_peered":"2024-03-08T08:29:06.244712+0000","last_unstale":"2024-03-08T08:29:07.235098+0000","last_undegraded":"2024-03-08T08:29:07.235098+0000","last_fullsized":"2024-03-08T08:29:07.235098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:23.178032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.260508+0000","last_change":"2024-03-08T08:29:06.255239+0000","last_active":"2024-03-08T08:29:06.260508+0000","last_peered":"2024-03-08T08:29:06.260508+0000","last_clean":"2024-03-08T08:29:06.260508+0000","last_became_active":"2024-03-08T08:29:06.254635+0000","last_became_peered":"2024-03-08T08:29:06.254635+0000","last_unstale":"2024-03-08T08:29:06.260508+0000","last_undegraded":"2024-03-08T08:29:06.260508+0000","last_fullsized":"2024-03-08T08:29:06.260508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:12.314555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236746+0000","last_change":"2024-03-08T08:29:06.252849+0000","last_active":"2024-03-08T08:29:07.236746+0000","last_peered":"2024-03-08T08:29:07.236746+0000","last_clean":"2024-03-08T08:29:07.236746+0000","last_became_active":"2024-03-08T08:29:06.252559+0000","last_became_peered":"2024-03-08T08:29:06.252559+0000","last_unstale":"2024-03-08T08:29:07.236746+0000","last_undegraded":"2024-03-08T08:29:07.236746+0000","last_fullsized":"2024-03-08T08:29:07.236746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:06:45.449613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905459+0000","last_change":"2024-03-08T08:29:05.247749+0000","last_active":"2024-03-08T08:29:07.905459+0000","last_peered":"2024-03-08T08:29:07.905459+0000","last_clean":"2024-03-08T08:29:07.905459+0000","last_became_active":"2024-03-08T08:29:05.247492+0000","last_became_peered":"2024-03-08T08:29:05.247492+0000","last_unstale":"2024-03-08T08:29:07.905459+0000","last_undegraded":"2024-03-08T08:29:07.905459+0000","last_fullsized":"2024-03-08T08:29:07.905459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:14:48.392940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235447+0000","last_change":"2024-03-08T08:29:06.248631+0000","last_active":"2024-03-08T08:29:07.235447+0000","last_peered":"2024-03-08T08:29:07.235447+0000","last_clean":"2024-03-08T08:29:07.235447+0000","last_became_active":"2024-03-08T08:29:06.248337+0000","last_became_peered":"2024-03-08T08:29:06.248337+0000","last_unstale":"2024-03-08T08:29:07.235447+0000","last_undegraded":"2024-03-08T08:29:07.235447+0000","last_fullsized":"2024-03-08T08:29:07.235447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.259316+0000","last_change":"2024-03-08T08:29:05.256580+0000","last_active":"2024-03-08T08:29:06.259316+0000","last_peered":"2024-03-08T08:29:06.259316+0000","last_clean":"2024-03-08T08:29:06.259316+0000","last_became_active":"2024-03-08T08:29:05.256286+0000","last_became_peered":"2024-03-08T08:29:05.256286+0000","last_unstale":"2024-03-08T08:29:06.259316+0000","last_undegraded":"2024-03-08T08:29:06.259316+0000","last_fullsized":"2024-03-08T08:29:06.259316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409076+0000","last_change":"2024-03-08T08:29:05.253153+0000","last_active":"2024-03-08T08:29:07.409076+0000","last_peered":"2024-03-08T08:29:07.409076+0000","last_clean":"2024-03-08T08:29:07.409076+0000","last_became_active":"2024-03-08T08:29:05.252819+0000","last_became_peered":"2024-03-08T08:29:05.252819+0000","last_unstale":"2024-03-08T08:29:07.409076+0000","last_undegraded":"2024-03-08T08:29:07.409076+0000","last_fullsized":"2024-03-08T08:29:07.409076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905491+0000","last_change":"2024-03-08T08:29:06.249692+0000","last_active":"2024-03-08T08:29:07.905491+0000","last_peered":"2024-03-08T08:29:07.905491+0000","last_clean":"2024-03-08T08:29:07.905491+0000","last_became_active":"2024-03-08T08:29:06.248953+0000","last_became_peered":"2024-03-08T08:29:06.248953+0000","last_unstale":"2024-03-08T08:29:07.905491+0000","last_undegraded":"2024-03-08T08:29:07.905491+0000","last_fullsized":"2024-03-08T08:29:07.905491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:30:01.675268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.245459+0000","last_change":"2024-03-08T08:29:05.257702+0000","last_active":"2024-03-08T08:29:06.245459+0000","last_peered":"2024-03-08T08:29:06.245459+0000","last_clean":"2024-03-08T08:29:06.245459+0000","last_became_active":"2024-03-08T08:29:05.257406+0000","last_became_peered":"2024-03-08T08:29:05.257406+0000","last_unstale":"2024-03-08T08:29:06.245459+0000","last_undegraded":"2024-03-08T08:29:06.245459+0000","last_fullsized":"2024-03-08T08:29:06.245459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:43.430990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235273+0000","last_change":"2024-03-08T08:29:06.246139+0000","last_active":"2024-03-08T08:29:07.235273+0000","last_peered":"2024-03-08T08:29:07.235273+0000","last_clean":"2024-03-08T08:29:07.235273+0000","last_became_active":"2024-03-08T08:29:06.245660+0000","last_became_peered":"2024-03-08T08:29:06.245660+0000","last_unstale":"2024-03-08T08:29:07.235273+0000","last_undegraded":"2024-03-08T08:29:07.235273+0000","last_fullsized":"2024-03-08T08:29:07.235273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236862+0000","last_change":"2024-03-08T08:29:05.240557+0000","last_active":"2024-03-08T08:29:07.236862+0000","last_peered":"2024-03-08T08:29:07.236862+0000","last_clean":"2024-03-08T08:29:07.236862+0000","last_became_active":"2024-03-08T08:29:05.240233+0000","last_became_peered":"2024-03-08T08:29:05.240233+0000","last_unstale":"2024-03-08T08:29:07.236862+0000","last_undegraded":"2024-03-08T08:29:07.236862+0000","last_fullsized":"2024-03-08T08:29:07.236862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.261061+0000","last_change":"2024-03-08T08:29:06.258421+0000","last_active":"2024-03-08T08:29:06.261061+0000","last_peered":"2024-03-08T08:29:06.261061+0000","last_clean":"2024-03-08T08:29:06.261061+0000","last_became_active":"2024-03-08T08:29:06.258079+0000","last_became_peered":"2024-03-08T08:29:06.258079+0000","last_unstale":"2024-03-08T08:29:06.261061+0000","last_undegraded":"2024-03-08T08:29:06.261061+0000","last_fullsized":"2024-03-08T08:29:06.261061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:39.994369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562645+0000","last_change":"2024-03-08T08:29:05.251488+0000","last_active":"2024-03-08T08:29:07.562645+0000","last_peered":"2024-03-08T08:29:07.562645+0000","last_clean":"2024-03-08T08:29:07.562645+0000","last_became_active":"2024-03-08T08:29:05.251220+0000","last_became_peered":"2024-03-08T08:29:05.251220+0000","last_unstale":"2024-03-08T08:29:07.562645+0000","last_undegraded":"2024-03-08T08:29:07.562645+0000","last_fullsized":"2024-03-08T08:29:07.562645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905718+0000","last_change":"2024-03-08T08:29:06.257799+0000","last_active":"2024-03-08T08:29:07.905718+0000","last_peered":"2024-03-08T08:29:07.905718+0000","last_clean":"2024-03-08T08:29:07.905718+0000","last_became_active":"2024-03-08T08:29:06.257545+0000","last_became_peered":"2024-03-08T08:29:06.257545+0000","last_unstale":"2024-03-08T08:29:07.905718+0000","last_undegraded":"2024-03-08T08:29:07.905718+0000","last_fullsized":"2024-03-08T08:29:07.905718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:21:44.744902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236028+0000","last_change":"2024-03-08T08:29:05.240046+0000","last_active":"2024-03-08T08:29:07.236028+0000","last_peered":"2024-03-08T08:29:07.236028+0000","last_clean":"2024-03-08T08:29:07.236028+0000","last_became_active":"2024-03-08T08:29:05.239754+0000","last_became_peered":"2024-03-08T08:29:05.239754+0000","last_unstale":"2024-03-08T08:29:07.236028+0000","last_undegraded":"2024-03-08T08:29:07.236028+0000","last_fullsized":"2024-03-08T08:29:07.236028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264488+0000","last_change":"2024-03-08T08:29:06.258228+0000","last_active":"2024-03-08T08:29:06.264488+0000","last_peered":"2024-03-08T08:29:06.264488+0000","last_clean":"2024-03-08T08:29:06.264488+0000","last_became_active":"2024-03-08T08:29:06.256849+0000","last_became_peered":"2024-03-08T08:29:06.256849+0000","last_unstale":"2024-03-08T08:29:06.264488+0000","last_undegraded":"2024-03-08T08:29:06.264488+0000","last_fullsized":"2024-03-08T08:29:06.264488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:42.233464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264787+0000","last_change":"2024-03-08T08:29:05.246882+0000","last_active":"2024-03-08T08:29:06.264787+0000","last_peered":"2024-03-08T08:29:06.264787+0000","last_clean":"2024-03-08T08:29:06.264787+0000","last_became_active":"2024-03-08T08:29:05.240261+0000","last_became_peered":"2024-03-08T08:29:05.240261+0000","last_unstale":"2024-03-08T08:29:06.264787+0000","last_undegraded":"2024-03-08T08:29:06.264787+0000","last_fullsized":"2024-03-08T08:29:06.264787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:20:06.037325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562506+0000","last_change":"2024-03-08T08:29:06.246529+0000","last_active":"2024-03-08T08:29:07.562506+0000","last_peered":"2024-03-08T08:29:07.562506+0000","last_clean":"2024-03-08T08:29:07.562506+0000","last_became_active":"2024-03-08T08:29:06.246230+0000","last_became_peered":"2024-03-08T08:29:06.246230+0000","last_unstale":"2024-03-08T08:29:07.562506+0000","last_undegraded":"2024-03-08T08:29:07.562506+0000","last_fullsized":"2024-03-08T08:29:07.562506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:23:31.113352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235779+0000","last_change":"2024-03-08T08:29:05.251723+0000","last_active":"2024-03-08T08:29:07.235779+0000","last_peered":"2024-03-08T08:29:07.235779+0000","last_clean":"2024-03-08T08:29:07.235779+0000","last_became_active":"2024-03-08T08:29:05.251410+0000","last_became_peered":"2024-03-08T08:29:05.251410+0000","last_unstale":"2024-03-08T08:29:07.235779+0000","last_undegraded":"2024-03-08T08:29:07.235779+0000","last_fullsized":"2024-03-08T08:29:07.235779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409264+0000","last_change":"2024-03-08T08:29:06.249661+0000","last_active":"2024-03-08T08:29:07.409264+0000","last_peered":"2024-03-08T08:29:07.409264+0000","last_clean":"2024-03-08T08:29:07.409264+0000","last_became_active":"2024-03-08T08:29:06.249388+0000","last_became_peered":"2024-03-08T08:29:06.249388+0000","last_unstale":"2024-03-08T08:29:07.409264+0000","last_undegraded":"2024-03-08T08:29:07.409264+0000","last_fullsized":"2024-03-08T08:29:07.409264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.243316+0000","last_change":"2024-03-08T08:29:05.250954+0000","last_active":"2024-03-08T08:29:06.243316+0000","last_peered":"2024-03-08T08:29:06.243316+0000","last_clean":"2024-03-08T08:29:06.243316+0000","last_became_active":"2024-03-08T08:29:05.250478+0000","last_became_peered":"2024-03-08T08:29:05.250478+0000","last_unstale":"2024-03-08T08:29:06.243316+0000","last_undegraded":"2024-03-08T08:29:06.243316+0000","last_fullsized":"2024-03-08T08:29:06.243316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:22:33.853562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.265398+0000","last_change":"2024-03-08T08:29:06.263756+0000","last_active":"2024-03-08T08:29:06.265398+0000","last_peered":"2024-03-08T08:29:06.265398+0000","last_clean":"2024-03-08T08:29:06.265398+0000","last_became_active":"2024-03-08T08:29:06.263461+0000","last_became_peered":"2024-03-08T08:29:06.263461+0000","last_unstale":"2024-03-08T08:29:06.265398+0000","last_undegraded":"2024-03-08T08:29:06.265398+0000","last_fullsized":"2024-03-08T08:29:06.265398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:40:11.278068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562192+0000","last_change":"2024-03-08T08:29:05.253842+0000","last_active":"2024-03-08T08:29:07.562192+0000","last_peered":"2024-03-08T08:29:07.562192+0000","last_clean":"2024-03-08T08:29:07.562192+0000","last_became_active":"2024-03-08T08:29:05.253552+0000","last_became_peered":"2024-03-08T08:29:05.253552+0000","last_unstale":"2024-03-08T08:29:07.562192+0000","last_undegraded":"2024-03-08T08:29:07.562192+0000","last_fullsized":"2024-03-08T08:29:07.562192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236519+0000","last_change":"2024-03-08T08:29:06.259240+0000","last_active":"2024-03-08T08:29:07.236519+0000","last_peered":"2024-03-08T08:29:07.236519+0000","last_clean":"2024-03-08T08:29:07.236519+0000","last_became_active":"2024-03-08T08:29:06.257182+0000","last_became_peered":"2024-03-08T08:29:06.257182+0000","last_unstale":"2024-03-08T08:29:07.236519+0000","last_undegraded":"2024-03-08T08:29:07.236519+0000","last_fullsized":"2024-03-08T08:29:07.236519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:58:34.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235055+0000","last_change":"2024-03-08T08:29:05.237318+0000","last_active":"2024-03-08T08:29:07.235055+0000","last_peered":"2024-03-08T08:29:07.235055+0000","last_clean":"2024-03-08T08:29:07.235055+0000","last_became_active":"2024-03-08T08:29:05.236985+0000","last_became_peered":"2024-03-08T08:29:05.236985+0000","last_unstale":"2024-03-08T08:29:07.235055+0000","last_undegraded":"2024-03-08T08:29:07.235055+0000","last_fullsized":"2024-03-08T08:29:07.235055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.261967+0000","last_change":"2024-03-08T08:29:06.256312+0000","last_active":"2024-03-08T08:29:06.261967+0000","last_peered":"2024-03-08T08:29:06.261967+0000","last_clean":"2024-03-08T08:29:06.261967+0000","last_became_active":"2024-03-08T08:29:06.256021+0000","last_became_peered":"2024-03-08T08:29:06.256021+0000","last_unstale":"2024-03-08T08:29:06.261967+0000","last_undegraded":"2024-03-08T08:29:06.261967+0000","last_fullsized":"2024-03-08T08:29:06.261967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562837+0000","last_change":"2024-03-08T08:29:05.253285+0000","last_active":"2024-03-08T08:29:07.562837+0000","last_peered":"2024-03-08T08:29:07.562837+0000","last_clean":"2024-03-08T08:29:07.562837+0000","last_became_active":"2024-03-08T08:29:05.253023+0000","last_became_peered":"2024-03-08T08:29:05.253023+0000","last_unstale":"2024-03-08T08:29:07.562837+0000","last_undegraded":"2024-03-08T08:29:07.562837+0000","last_fullsized":"2024-03-08T08:29:07.562837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264730+0000","last_change":"2024-03-08T08:29:06.263316+0000","last_active":"2024-03-08T08:29:06.264730+0000","last_peered":"2024-03-08T08:29:06.264730+0000","last_clean":"2024-03-08T08:29:06.264730+0000","last_became_active":"2024-03-08T08:29:06.263033+0000","last_became_peered":"2024-03-08T08:29:06.263033+0000","last_unstale":"2024-03-08T08:29:06.264730+0000","last_undegraded":"2024-03-08T08:29:06.264730+0000","last_fullsized":"2024-03-08T08:29:06.264730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:29.316489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.247430+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.247146+0000","last_became_peered":"2024-03-08T08:29:05.247146+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562783+0000","last_change":"2024-03-08T08:29:06.246687+0000","last_active":"2024-03-08T08:29:07.562783+0000","last_peered":"2024-03-08T08:29:07.562783+0000","last_clean":"2024-03-08T08:29:07.562783+0000","last_became_active":"2024-03-08T08:29:06.246421+0000","last_became_peered":"2024-03-08T08:29:06.246421+0000","last_unstale":"2024-03-08T08:29:07.562783+0000","last_undegraded":"2024-03-08T08:29:07.562783+0000","last_fullsized":"2024-03-08T08:29:07.562783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:47:29.808023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905309+0000","last_change":"2024-03-08T08:29:05.246842+0000","last_active":"2024-03-08T08:29:07.905309+0000","last_peered":"2024-03-08T08:29:07.905309+0000","last_clean":"2024-03-08T08:29:07.905309+0000","last_became_active":"2024-03-08T08:29:05.246589+0000","last_became_peered":"2024-03-08T08:29:05.246589+0000","last_unstale":"2024-03-08T08:29:07.905309+0000","last_undegraded":"2024-03-08T08:29:07.905309+0000","last_fullsized":"2024-03-08T08:29:07.905309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:34:13.125245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562749+0000","last_change":"2024-03-08T08:29:06.250869+0000","last_active":"2024-03-08T08:29:07.562749+0000","last_peered":"2024-03-08T08:29:07.562749+0000","last_clean":"2024-03-08T08:29:07.562749+0000","last_became_active":"2024-03-08T08:29:06.250598+0000","last_became_peered":"2024-03-08T08:29:06.250598+0000","last_unstale":"2024-03-08T08:29:07.562749+0000","last_undegraded":"2024-03-08T08:29:07.562749+0000","last_fullsized":"2024-03-08T08:29:07.562749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:37:49.534588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409554+0000","last_change":"2024-03-08T08:29:05.253164+0000","last_active":"2024-03-08T08:29:07.409554+0000","last_peered":"2024-03-08T08:29:07.409554+0000","last_clean":"2024-03-08T08:29:07.409554+0000","last_became_active":"2024-03-08T08:29:05.252833+0000","last_became_peered":"2024-03-08T08:29:05.252833+0000","last_unstale":"2024-03-08T08:29:07.409554+0000","last_undegraded":"2024-03-08T08:29:07.409554+0000","last_fullsized":"2024-03-08T08:29:07.409554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264736+0000","last_change":"2024-03-08T08:29:06.259015+0000","last_active":"2024-03-08T08:29:06.264736+0000","last_peered":"2024-03-08T08:29:06.264736+0000","last_clean":"2024-03-08T08:29:06.264736+0000","last_became_active":"2024-03-08T08:29:06.258632+0000","last_became_peered":"2024-03-08T08:29:06.258632+0000","last_unstale":"2024-03-08T08:29:06.264736+0000","last_undegraded":"2024-03-08T08:29:06.264736+0000","last_fullsized":"2024-03-08T08:29:06.264736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:52:02.676611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236450+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:07.236450+0000","last_peered":"2024-03-08T08:29:07.236450+0000","last_clean":"2024-03-08T08:29:07.236450+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:07.236450+0000","last_undegraded":"2024-03-08T08:29:07.236450+0000","last_fullsized":"2024-03-08T08:29:07.236450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.245179+0000","last_change":"2024-03-08T08:29:05.257931+0000","last_active":"2024-03-08T08:29:06.245179+0000","last_peered":"2024-03-08T08:29:06.245179+0000","last_clean":"2024-03-08T08:29:06.245179+0000","last_became_active":"2024-03-08T08:29:05.257637+0000","last_became_peered":"2024-03-08T08:29:05.257637+0000","last_unstale":"2024-03-08T08:29:06.245179+0000","last_undegraded":"2024-03-08T08:29:06.245179+0000","last_fullsized":"2024-03-08T08:29:06.245179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:00.694258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235934+0000","last_change":"2024-03-08T08:29:06.251166+0000","last_active":"2024-03-08T08:29:07.235934+0000","last_peered":"2024-03-08T08:29:07.235934+0000","last_clean":"2024-03-08T08:29:07.235934+0000","last_became_active":"2024-03-08T08:29:06.250778+0000","last_became_peered":"2024-03-08T08:29:06.250778+0000","last_unstale":"2024-03-08T08:29:07.235934+0000","last_undegraded":"2024-03-08T08:29:07.235934+0000","last_fullsized":"2024-03-08T08:29:07.235934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:14:53.496011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.245622+0000","last_change":"2024-03-08T08:29:05.251244+0000","last_active":"2024-03-08T08:29:06.245622+0000","last_peered":"2024-03-08T08:29:06.245622+0000","last_clean":"2024-03-08T08:29:06.245622+0000","last_became_active":"2024-03-08T08:29:05.250928+0000","last_became_peered":"2024-03-08T08:29:05.250928+0000","last_unstale":"2024-03-08T08:29:06.245622+0000","last_undegraded":"2024-03-08T08:29:06.245622+0000","last_fullsized":"2024-03-08T08:29:06.245622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:52.269276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409152+0000","last_change":"2024-03-08T08:29:06.255405+0000","last_active":"2024-03-08T08:29:07.409152+0000","last_peered":"2024-03-08T08:29:07.409152+0000","last_clean":"2024-03-08T08:29:07.409152+0000","last_became_active":"2024-03-08T08:29:06.255118+0000","last_became_peered":"2024-03-08T08:29:06.255118+0000","last_unstale":"2024-03-08T08:29:07.409152+0000","last_undegraded":"2024-03-08T08:29:07.409152+0000","last_fullsized":"2024-03-08T08:29:07.409152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:44.436118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235211+0000","last_change":"2024-03-08T08:29:05.245447+0000","last_active":"2024-03-08T08:29:07.235211+0000","last_peered":"2024-03-08T08:29:07.235211+0000","last_clean":"2024-03-08T08:29:07.235211+0000","last_became_active":"2024-03-08T08:29:05.245148+0000","last_became_peered":"2024-03-08T08:29:05.245148+0000","last_unstale":"2024-03-08T08:29:07.235211+0000","last_undegraded":"2024-03-08T08:29:07.235211+0000","last_fullsized":"2024-03-08T08:29:07.235211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905516+0000","last_change":"2024-03-08T08:29:06.248546+0000","last_active":"2024-03-08T08:29:07.905516+0000","last_peered":"2024-03-08T08:29:07.905516+0000","last_clean":"2024-03-08T08:29:07.905516+0000","last_became_active":"2024-03-08T08:29:06.248292+0000","last_became_peered":"2024-03-08T08:29:06.248292+0000","last_unstale":"2024-03-08T08:29:07.905516+0000","last_undegraded":"2024-03-08T08:29:07.905516+0000","last_fullsized":"2024-03-08T08:29:07.905516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:32:56.591618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.243444+0000","last_change":"2024-03-08T08:29:05.250674+0000","last_active":"2024-03-08T08:29:06.243444+0000","last_peered":"2024-03-08T08:29:06.243444+0000","last_clean":"2024-03-08T08:29:06.243444+0000","last_became_active":"2024-03-08T08:29:05.250342+0000","last_became_peered":"2024-03-08T08:29:05.250342+0000","last_unstale":"2024-03-08T08:29:06.243444+0000","last_undegraded":"2024-03-08T08:29:06.243444+0000","last_fullsized":"2024-03-08T08:29:06.243444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:04.924475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562458+0000","last_change":"2024-03-08T08:29:06.252736+0000","last_active":"2024-03-08T08:29:07.562458+0000","last_peered":"2024-03-08T08:29:07.562458+0000","last_clean":"2024-03-08T08:29:07.562458+0000","last_became_active":"2024-03-08T08:29:06.251566+0000","last_became_peered":"2024-03-08T08:29:06.251566+0000","last_unstale":"2024-03-08T08:29:07.562458+0000","last_undegraded":"2024-03-08T08:29:07.562458+0000","last_fullsized":"2024-03-08T08:29:07.562458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:52:02.676611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905374+0000","last_change":"2024-03-08T08:29:05.245146+0000","last_active":"2024-03-08T08:29:07.905374+0000","last_peered":"2024-03-08T08:29:07.905374+0000","last_clean":"2024-03-08T08:29:07.905374+0000","last_became_active":"2024-03-08T08:29:05.244812+0000","last_became_peered":"2024-03-08T08:29:05.244812+0000","last_unstale":"2024-03-08T08:29:07.905374+0000","last_undegraded":"2024-03-08T08:29:07.905374+0000","last_fullsized":"2024-03-08T08:29:07.905374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.260250+0000","last_change":"2024-03-08T08:29:06.254952+0000","last_active":"2024-03-08T08:29:06.260250+0000","last_peered":"2024-03-08T08:29:06.260250+0000","last_clean":"2024-03-08T08:29:06.260250+0000","last_became_active":"2024-03-08T08:29:06.254408+0000","last_became_peered":"2024-03-08T08:29:06.254408+0000","last_unstale":"2024-03-08T08:29:06.260250+0000","last_undegraded":"2024-03-08T08:29:06.260250+0000","last_fullsized":"2024-03-08T08:29:06.260250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:21.083802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.240976+0000","last_change":"2024-03-08T08:29:05.256587+0000","last_active":"2024-03-08T08:29:06.240976+0000","last_peered":"2024-03-08T08:29:06.240976+0000","last_clean":"2024-03-08T08:29:06.240976+0000","last_became_active":"2024-03-08T08:29:05.256091+0000","last_became_peered":"2024-03-08T08:29:05.256091+0000","last_unstale":"2024-03-08T08:29:06.240976+0000","last_undegraded":"2024-03-08T08:29:06.240976+0000","last_fullsized":"2024-03-08T08:29:06.240976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409456+0000","last_change":"2024-03-08T08:29:06.250994+0000","last_active":"2024-03-08T08:29:07.409456+0000","last_peered":"2024-03-08T08:29:07.409456+0000","last_clean":"2024-03-08T08:29:07.409456+0000","last_became_active":"2024-03-08T08:29:06.250738+0000","last_became_peered":"2024-03-08T08:29:06.250738+0000","last_unstale":"2024-03-08T08:29:07.409456+0000","last_undegraded":"2024-03-08T08:29:07.409456+0000","last_fullsized":"2024-03-08T08:29:07.409456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:08.239687+0000","last_change":"2024-03-08T08:29:05.251705+0000","last_active":"2024-03-08T08:29:08.239687+0000","last_peered":"2024-03-08T08:29:08.239687+0000","last_clean":"2024-03-08T08:29:08.239687+0000","last_became_active":"2024-03-08T08:29:05.251424+0000","last_became_peered":"2024-03-08T08:29:05.251424+0000","last_unstale":"2024-03-08T08:29:08.239687+0000","last_undegraded":"2024-03-08T08:29:08.239687+0000","last_fullsized":"2024-03-08T08:29:08.239687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235787+0000","last_change":"2024-03-08T08:29:06.248049+0000","last_active":"2024-03-08T08:29:07.235787+0000","last_peered":"2024-03-08T08:29:07.235787+0000","last_clean":"2024-03-08T08:29:07.235787+0000","last_became_active":"2024-03-08T08:29:06.243640+0000","last_became_peered":"2024-03-08T08:29:06.243640+0000","last_unstale":"2024-03-08T08:29:07.235787+0000","last_undegraded":"2024-03-08T08:29:07.235787+0000","last_fullsized":"2024-03-08T08:29:07.235787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:18:25.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.263152+0000","last_change":"2024-03-08T08:29:05.256862+0000","last_active":"2024-03-08T08:29:06.263152+0000","last_peered":"2024-03-08T08:29:06.263152+0000","last_clean":"2024-03-08T08:29:06.263152+0000","last_became_active":"2024-03-08T08:29:05.256474+0000","last_became_peered":"2024-03-08T08:29:05.256474+0000","last_unstale":"2024-03-08T08:29:06.263152+0000","last_undegraded":"2024-03-08T08:29:06.263152+0000","last_fullsized":"2024-03-08T08:29:06.263152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235404+0000","last_change":"2024-03-08T08:29:06.240193+0000","last_active":"2024-03-08T08:29:07.235404+0000","last_peered":"2024-03-08T08:29:07.235404+0000","last_clean":"2024-03-08T08:29:07.235404+0000","last_became_active":"2024-03-08T08:29:06.239901+0000","last_became_peered":"2024-03-08T08:29:06.239901+0000","last_unstale":"2024-03-08T08:29:07.235404+0000","last_undegraded":"2024-03-08T08:29:07.235404+0000","last_fullsized":"2024-03-08T08:29:07.235404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562137+0000","last_change":"2024-03-08T08:29:05.250815+0000","last_active":"2024-03-08T08:29:07.562137+0000","last_peered":"2024-03-08T08:29:07.562137+0000","last_clean":"2024-03-08T08:29:07.562137+0000","last_became_active":"2024-03-08T08:29:05.250443+0000","last_became_peered":"2024-03-08T08:29:05.250443+0000","last_unstale":"2024-03-08T08:29:07.562137+0000","last_undegraded":"2024-03-08T08:29:07.562137+0000","last_fullsized":"2024-03-08T08:29:07.562137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.256683+0000","last_change":"2024-03-08T08:29:06.253615+0000","last_active":"2024-03-08T08:29:06.256683+0000","last_peered":"2024-03-08T08:29:06.256683+0000","last_clean":"2024-03-08T08:29:06.256683+0000","last_became_active":"2024-03-08T08:29:06.253324+0000","last_became_peered":"2024-03-08T08:29:06.253324+0000","last_unstale":"2024-03-08T08:29:06.256683+0000","last_undegraded":"2024-03-08T08:29:06.256683+0000","last_fullsized":"2024-03-08T08:29:06.256683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.251684+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.251395+0000","last_became_peered":"2024-03-08T08:29:05.251395+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235017+0000","last_change":"2024-03-08T08:29:06.237785+0000","last_active":"2024-03-08T08:29:07.235017+0000","last_peered":"2024-03-08T08:29:07.235017+0000","last_clean":"2024-03-08T08:29:07.235017+0000","last_became_active":"2024-03-08T08:29:06.237449+0000","last_became_peered":"2024-03-08T08:29:06.237449+0000","last_unstale":"2024-03-08T08:29:07.235017+0000","last_undegraded":"2024-03-08T08:29:07.235017+0000","last_fullsized":"2024-03-08T08:29:07.235017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409279+0000","last_change":"2024-03-08T08:29:05.246589+0000","last_active":"2024-03-08T08:29:07.409279+0000","last_peered":"2024-03-08T08:29:07.409279+0000","last_clean":"2024-03-08T08:29:07.409279+0000","last_became_active":"2024-03-08T08:29:05.246204+0000","last_became_peered":"2024-03-08T08:29:05.246204+0000","last_unstale":"2024-03-08T08:29:07.409279+0000","last_undegraded":"2024-03-08T08:29:07.409279+0000","last_fullsized":"2024-03-08T08:29:07.409279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235180+0000","last_change":"2024-03-08T08:29:06.244533+0000","last_active":"2024-03-08T08:29:07.235180+0000","last_peered":"2024-03-08T08:29:07.235180+0000","last_clean":"2024-03-08T08:29:07.235180+0000","last_became_active":"2024-03-08T08:29:06.244232+0000","last_became_peered":"2024-03-08T08:29:06.244232+0000","last_unstale":"2024-03-08T08:29:07.235180+0000","last_undegraded":"2024-03-08T08:29:07.235180+0000","last_fullsized":"2024-03-08T08:29:07.235180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:36.028927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.257249+0000","last_change":"2024-03-08T08:29:05.258093+0000","last_active":"2024-03-08T08:29:06.257249+0000","last_peered":"2024-03-08T08:29:06.257249+0000","last_clean":"2024-03-08T08:29:06.257249+0000","last_became_active":"2024-03-08T08:29:05.257817+0000","last_became_peered":"2024-03-08T08:29:05.257817+0000","last_unstale":"2024-03-08T08:29:06.257249+0000","last_undegraded":"2024-03-08T08:29:06.257249+0000","last_fullsized":"2024-03-08T08:29:06.257249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:18:30.428380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264892+0000","last_change":"2024-03-08T08:29:06.258011+0000","last_active":"2024-03-08T08:29:06.264892+0000","last_peered":"2024-03-08T08:29:06.264892+0000","last_clean":"2024-03-08T08:29:06.264892+0000","last_became_active":"2024-03-08T08:29:06.257742+0000","last_became_peered":"2024-03-08T08:29:06.257742+0000","last_unstale":"2024-03-08T08:29:06.264892+0000","last_undegraded":"2024-03-08T08:29:06.264892+0000","last_fullsized":"2024-03-08T08:29:06.264892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:16:24.171714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-08T08:29:06.264774+0000","last_change":"2024-03-08T08:29:06.262040+0000","last_active":"2024-03-08T08:29:06.264774+0000","last_peered":"2024-03-08T08:29:06.264774+0000","last_clean":"2024-03-08T08:29:06.264774+0000","last_became_active":"2024-03-08T08:29:06.258293+0000","last_became_peered":"2024-03-08T08:29:06.258293+0000","last_unstale":"2024-03-08T08:29:06.264774+0000","last_undegraded":"2024-03-08T08:29:06.264774+0000","last_fullsized":"2024-03-08T08:29:06.264774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:47:43.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562776+0000","last_change":"2024-03-08T08:29:05.252744+0000","last_active":"2024-03-08T08:29:07.562776+0000","last_peered":"2024-03-08T08:29:07.562776+0000","last_clean":"2024-03-08T08:29:07.562776+0000","last_became_active":"2024-03-08T08:29:05.252365+0000","last_became_peered":"2024-03-08T08:29:05.252365+0000","last_unstale":"2024-03-08T08:29:07.562776+0000","last_undegraded":"2024-03-08T08:29:07.562776+0000","last_fullsized":"2024-03-08T08:29:07.562776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236776+0000","last_change":"2024-03-08T08:29:06.260541+0000","last_active":"2024-03-08T08:29:07.236776+0000","last_peered":"2024-03-08T08:29:07.236776+0000","last_clean":"2024-03-08T08:29:07.236776+0000","last_became_active":"2024-03-08T08:29:06.260283+0000","last_became_peered":"2024-03-08T08:29:06.260283+0000","last_unstale":"2024-03-08T08:29:07.236776+0000","last_undegraded":"2024-03-08T08:29:07.236776+0000","last_fullsized":"2024-03-08T08:29:07.236776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:52:33.000979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562608+0000","last_change":"2024-03-08T08:29:05.251622+0000","last_active":"2024-03-08T08:29:07.562608+0000","last_peered":"2024-03-08T08:29:07.562608+0000","last_clean":"2024-03-08T08:29:07.562608+0000","last_became_active":"2024-03-08T08:29:05.251346+0000","last_became_peered":"2024-03-08T08:29:05.251346+0000","last_unstale":"2024-03-08T08:29:07.562608+0000","last_undegraded":"2024-03-08T08:29:07.562608+0000","last_fullsized":"2024-03-08T08:29:07.562608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905651+0000","last_change":"2024-03-08T08:29:06.245002+0000","last_active":"2024-03-08T08:29:07.905651+0000","last_peered":"2024-03-08T08:29:07.905651+0000","last_clean":"2024-03-08T08:29:07.905651+0000","last_became_active":"2024-03-08T08:29:06.244765+0000","last_became_peered":"2024-03-08T08:29:06.244765+0000","last_unstale":"2024-03-08T08:29:07.905651+0000","last_undegraded":"2024-03-08T08:29:07.905651+0000","last_fullsized":"2024-03-08T08:29:07.905651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:14:06.939392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905631+0000","last_change":"2024-03-08T08:29:05.245332+0000","last_active":"2024-03-08T08:29:07.905631+0000","last_peered":"2024-03-08T08:29:07.905631+0000","last_clean":"2024-03-08T08:29:07.905631+0000","last_became_active":"2024-03-08T08:29:05.245042+0000","last_became_peered":"2024-03-08T08:29:05.245042+0000","last_unstale":"2024-03-08T08:29:07.905631+0000","last_undegraded":"2024-03-08T08:29:07.905631+0000","last_fullsized":"2024-03-08T08:29:07.905631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:36:06.948887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905465+0000","last_change":"2024-03-08T08:29:06.251546+0000","last_active":"2024-03-08T08:29:07.905465+0000","last_peered":"2024-03-08T08:29:07.905465+0000","last_clean":"2024-03-08T08:29:07.905465+0000","last_became_active":"2024-03-08T08:29:06.251278+0000","last_became_peered":"2024-03-08T08:29:06.251278+0000","last_unstale":"2024-03-08T08:29:07.905465+0000","last_undegraded":"2024-03-08T08:29:07.905465+0000","last_fullsized":"2024-03-08T08:29:07.905465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:29:28.214850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235915+0000","last_change":"2024-03-08T08:29:05.253532+0000","last_active":"2024-03-08T08:29:07.235915+0000","last_peered":"2024-03-08T08:29:07.235915+0000","last_clean":"2024-03-08T08:29:07.235915+0000","last_became_active":"2024-03-08T08:29:05.252904+0000","last_became_peered":"2024-03-08T08:29:05.252904+0000","last_unstale":"2024-03-08T08:29:07.235915+0000","last_undegraded":"2024-03-08T08:29:07.235915+0000","last_fullsized":"2024-03-08T08:29:07.235915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":73883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36144,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335696,"statfs":{"total":96636764160,"available":96599752704,"internally_reserved":0,"allocated":815104,"data_stored":664343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":76167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:13.996 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-08T08:29:13.997 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-08T08:29:13.997 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-08T08:29:13.997 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-08T08:29:13.997 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-08T08:29:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.133+0000 7fddad6eb640 1 Processor -- start 2024-03-08T08:29:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.133+0000 7fddad6eb640 1 -- start start 2024-03-08T08:29:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fddad6eb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdda80d34c0 0x7fdda81688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fddad6eb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda815e930 con 0x7fdda80b9420 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fddad6eb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda815ebc0 con 0x7fdda806c240 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fddad6eb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda80b6b30 con 0x7fdda80d34c0 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdda80d34c0 0x7fdda81688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdda80d34c0 0x7fdda81688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57746/0 (socket says 172.21.15.12:57746) 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 -- 172.21.15.12:0/1909978598 learned_addr learned my addr 172.21.15.12:0/1909978598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdd9ffff640 1 --1- 172.21.15.12:0/1909978598 >> v1:172.21.15.12:6789/0 conn(0x7fdda80b9420 0x7fdda815e220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52766/0 (socket says 172.21.15.12:52766) 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 -- 172.21.15.12:0/1909978598 >> v1:172.21.15.156:6789/0 conn(0x7fdda806c240 legacy=0x7fdda806c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 -- 172.21.15.12:0/1909978598 >> v1:172.21.15.12:6789/0 conn(0x7fdda80b9420 legacy=0x7fdda815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda77fe640 1 -- 172.21.15.12:0/1909978598 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda815eea0 con 0x7fdda80d34c0 2024-03-08T08:29:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.134+0000 7fdda67fc640 1 -- 172.21.15.12:0/1909978598 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535138056 0 0) 0x7fdda815e930 con 0x7fdda80b9420 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fdda77fe640 1 --2- 172.21.15.12:0/1909978598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80d34c0 0x7fdda81688a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e50 tx=0x7fdda002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1052b7c10f846f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fdda67fc640 1 -- 172.21.15.12:0/1909978598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda002cbb0 con 0x7fdda80d34c0 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fdda67fc640 1 -- 172.21.15.12:0/1909978598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda002cd50 con 0x7fdda80d34c0 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fdda67fc640 1 -- 172.21.15.12:0/1909978598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0034610 con 0x7fdda80d34c0 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fddad6eb640 1 -- 172.21.15.12:0/1909978598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80d34c0 msgr2=0x7fdda81688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.135+0000 7fddad6eb640 1 --2- 172.21.15.12:0/1909978598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80d34c0 0x7fdda81688a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e50 tx=0x7fdda002c010 comp rx=0 tx=0).stop 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.136+0000 7fddad6eb640 1 -- 172.21.15.12:0/1909978598 shutdown_connections 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.136+0000 7fddad6eb640 1 --2- 172.21.15.12:0/1909978598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80d34c0 0x7fdda81688a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.136+0000 7fddad6eb640 1 -- 172.21.15.12:0/1909978598 >> 172.21.15.12:0/1909978598 conn(0x7fdda8072900 msgr2=0x7fdda80d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.136+0000 7fddad6eb640 1 -- 172.21.15.12:0/1909978598 shutdown_connections 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.136+0000 7fddad6eb640 1 -- 172.21.15.12:0/1909978598 wait complete. 2024-03-08T08:29:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 Processor -- start 2024-03-08T08:29:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 -- start start 2024-03-08T08:29:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdda806c240 0x7fdda80cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdda80d34c0 0x7fdda815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda815bc30 con 0x7fdda80b9420 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80d1f90 con 0x7fdda806c240 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.137+0000 7fddad6eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda815f180 con 0x7fdda80d34c0 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57758/0 (socket says 172.21.15.12:57758) 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 -- 172.21.15.12:0/3421645859 learned_addr learned my addr 172.21.15.12:0/3421645859 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdda6ffd640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdda806c240 0x7fdda80cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdda77fe640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdda80d34c0 0x7fdda815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 -- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdda80d34c0 msgr2=0x7fdda815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdda80d34c0 0x7fdda815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 -- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdda806c240 msgr2=0x7fdda80cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdda806c240 0x7fdda80cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 -- 172.21.15.12:0/3421645859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda80b6210 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9ffff640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000c4e0 tx=0x7fdd9000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90018070 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd90007500 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90004a20 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda80b4720 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.138+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda80b4ce0 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.139+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda80749c0 con 0x7fdda80b9420 2024-03-08T08:29:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.140+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd9001b070 con 0x7fdda80b9420 2024-03-08T08:29:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.140+0000 7fdd9f7fe640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.140+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd900894b0 con 0x7fdda80b9420 2024-03-08T08:29:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.140+0000 7fdda6ffd640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.141+0000 7fdda6ffd640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 0x7fdd8806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004f60 tx=0x7fdd98007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.144+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdda80b6210 con 0x7fdda80b9420 2024-03-08T08:29:14.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:29:14.376 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:29:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.379+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdda8160f90 con 0x7fdda80b9420 2024-03-08T08:29:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.382+0000 7fdd9f7fe640 1 -- 172.21.15.12:0/3421645859 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171784 (secure 0 0 0) 0x7fdd90057da0 con 0x7fdda80b9420 2024-03-08T08:29:14.385 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 msgr2=0x7fdd8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 0x7fdd8806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004f60 tx=0x7fdd98007e30 comp rx=0 tx=0).stop 2024-03-08T08:29:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 msgr2=0x7fdda80b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000c4e0 tx=0x7fdd9000c9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 shutdown_connections 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdda80d34c0 0x7fdda815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdda80b9420 0x7fdda80b5cd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 --2- 172.21.15.12:0/3421645859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdda806c240 0x7fdda80cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 >> 172.21.15.12:0/3421645859 conn(0x7fdda8072900 msgr2=0x7fdda80b8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 shutdown_connections 2024-03-08T08:29:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.384+0000 7fddad6eb640 1 -- 172.21.15.12:0/3421645859 wait complete. 2024-03-08T08:29:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2068370000},{"type":"v1","addr":"172.21.15.12:6833","nonce":2068370000}]},"active_addr":"172.21.15.12:6833/2068370000","active_change":"2024-03-08T08:28:52.986603+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1846352590}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":528182448}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1574328370}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1747263207}]}]} 2024-03-08T08:29:14.399 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-08T08:29:14.399 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-08T08:29:14.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.566+0000 7fa1f28cd640 1 Processor -- start 2024-03-08T08:29:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.566+0000 7fa1f28cd640 1 -- start start 2024-03-08T08:29:14.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1ec0aef50 0x7fa1ec0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa1ec070140 0x7fa1ec15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec06e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0af330 con 0x7fa1ec0aef50 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec15e270 con 0x7fa1ec070140 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1f28cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b5610 con 0x7fa1ec15ef50 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1ec0aef50 0x7fa1ec0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1ec0aef50 0x7fa1ec0b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57764/0 (socket says 172.21.15.12:57764) 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 -- 172.21.15.12:0/2046273793 learned_addr learned my addr 172.21.15.12:0/2046273793 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 -- 172.21.15.12:0/2046273793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 msgr2=0x7fa1ec06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/2046273793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1eb7fe640 1 --2- 172.21.15.12:0/2046273793 >> v2:172.21.15.156:3300/0 conn(0x7fa1ec070140 0x7fa1ec15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 -- 172.21.15.12:0/2046273793 >> v2:172.21.15.156:3300/0 conn(0x7fa1ec070140 msgr2=0x7fa1ec15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/2046273793 >> v2:172.21.15.156:3300/0 conn(0x7fa1ec070140 0x7fa1ec15ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.567+0000 7fa1ebfff640 1 -- 172.21.15.12:0/2046273793 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0b58f0 con 0x7fa1ec0aef50 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.568+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/2046273793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec0b3910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0719d0 tx=0x7fa1e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96981ae95dbfaa03 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.568+0000 7fa1eaffd640 1 -- 172.21.15.12:0/2046273793 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0035020 con 0x7fa1ec0aef50 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.568+0000 7fa1eaffd640 1 -- 172.21.15.12:0/2046273793 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0031870 con 0x7fa1ec0aef50 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.568+0000 7fa1eaffd640 1 -- 172.21.15.12:0/2046273793 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0031c40 con 0x7fa1ec0aef50 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 -- 172.21.15.12:0/2046273793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 msgr2=0x7fa1ec0b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/2046273793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec0b3910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0719d0 tx=0x7fa1e002b910 comp rx=0 tx=0).stop 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 -- 172.21.15.12:0/2046273793 shutdown_connections 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/2046273793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/2046273793 >> v2:172.21.15.156:3300/0 conn(0x7fa1ec070140 0x7fa1ec15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/2046273793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec0b3910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 -- 172.21.15.12:0/2046273793 >> 172.21.15.12:0/2046273793 conn(0x7fa1ec0b5ca0 msgr2=0x7fa1ec0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 -- 172.21.15.12:0/2046273793 shutdown_connections 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.569+0000 7fa1f28cd640 1 -- 172.21.15.12:0/2046273793 wait complete. 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.570+0000 7fa1f28cd640 1 Processor -- start 2024-03-08T08:29:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.570+0000 7fa1f28cd640 1 -- start start 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 0x7fa1ec14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b5230 con 0x7fa1ec0aef50 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0705c0 con 0x7fa1ec070140 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f28cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b53b0 con 0x7fa1ec15ef50 2024-03-08T08:29:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57768/0 (socket says 172.21.15.12:57768) 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 -- 172.21.15.12:0/317413308 learned_addr learned my addr 172.21.15.12:0/317413308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 0x7fa1ec14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f0e6a640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 -- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 msgr2=0x7fa1ec0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec0b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 -- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 msgr2=0x7fa1ec14a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 0x7fa1ec14a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 -- 172.21.15.12:0/317413308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec152b90 con 0x7fa1ec0aef50 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 0x7fa1ec14a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1f0e6a640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec0b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1eb7fe640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c7f0 tx=0x7fa1dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.571+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc015070 con 0x7fa1ec0aef50 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.572+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc004220 con 0x7fa1ec0aef50 2024-03-08T08:29:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.572+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec152e70 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.572+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc007620 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.572+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec153430 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.573+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec0749c0 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.573+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1dc0079d0 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.573+0000 7fa1e8ff9640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.574+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.574+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1dc08a530 con 0x7fa1ec0aef50 2024-03-08T08:29:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.574+0000 7fa1ebfff640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0036000 tx=0x7fa1e0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.578+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa1dc059440 con 0x7fa1ec0aef50 2024-03-08T08:29:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.760+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1ec15b440 con 0x7fa1ec0aef50 2024-03-08T08:29:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.761+0000 7fa1e8ff9640 1 -- 172.21.15.12:0/317413308 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fa1dc058d40 con 0x7fa1ec0aef50 2024-03-08T08:29:14.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:14.763 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 msgr2=0x7fa1d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0036000 tx=0x7fa1e0008030 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 msgr2=0x7fa1ec152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c7f0 tx=0x7fa1dc00ccb0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 shutdown_connections 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1ec15ef50 0x7fa1ec0b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ec0aef50 0x7fa1ec152220 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 --2- 172.21.15.12:0/317413308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1ec070140 0x7fa1ec14a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.763+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 >> 172.21.15.12:0/317413308 conn(0x7fa1ec0b5ca0 msgr2=0x7fa1ec15d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.764+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 shutdown_connections 2024-03-08T08:29:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.764+0000 7fa1f28cd640 1 -- 172.21.15.12:0/317413308 wait complete. 2024-03-08T08:29:14.775 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-08T08:29:14.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.875+0000 7fe91ac9a640 1 Processor -- start 2024-03-08T08:29:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.875+0000 7fe91ac9a640 1 -- start start 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe91ac9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9140d1110 0x7fe9140b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe91ac9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe91ac9a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91415e1a0 con 0x7fe91415ddc0 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe91ac9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9140b9030 con 0x7fe9140d1110 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe91ac9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140d1d20 con 0x7fe9140d31c0 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe913fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9140d1110 0x7fe9140b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe918a36640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe91415ddc0 0x7fe9140d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52772/0 (socket says 172.21.15.12:52772) 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe918a36640 1 -- 172.21.15.12:0/2486603616 learned_addr learned my addr 172.21.15.12:0/2486603616 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.876+0000 7fe919237640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe913fff640 1 -- 172.21.15.12:0/2486603616 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 msgr2=0x7fe9140b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe913fff640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe913fff640 1 -- 172.21.15.12:0/2486603616 >> v1:172.21.15.12:6789/0 conn(0x7fe91415ddc0 legacy=0x7fe9140d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe913fff640 1 -- 172.21.15.12:0/2486603616 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140b8790 con 0x7fe9140d1110 2024-03-08T08:29:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe9137fe640 1 -- 172.21.15.12:0/2486603616 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039647057 0 0) 0x7fe91415e1a0 con 0x7fe91415ddc0 2024-03-08T08:29:14.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe913fff640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140b6be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe9000097a0 tx=0x7fe90002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335930b07bb31c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe9137fe640 1 -- 172.21.15.12:0/2486603616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900035020 con 0x7fe9140d1110 2024-03-08T08:29:14.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe9137fe640 1 -- 172.21.15.12:0/2486603616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900031810 con 0x7fe9140d1110 2024-03-08T08:29:14.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.877+0000 7fe9137fe640 1 -- 172.21.15.12:0/2486603616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900031be0 con 0x7fe9140d1110 2024-03-08T08:29:14.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 -- 172.21.15.12:0/2486603616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 msgr2=0x7fe9140b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140b6be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe9000097a0 tx=0x7fe90002b8b0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 -- 172.21.15.12:0/2486603616 shutdown_connections 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/2486603616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140b6be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 -- 172.21.15.12:0/2486603616 >> 172.21.15.12:0/2486603616 conn(0x7fe914072720 msgr2=0x7fe914072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 -- 172.21.15.12:0/2486603616 shutdown_connections 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.878+0000 7fe91ac9a640 1 -- 172.21.15.12:0/2486603616 wait complete. 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.879+0000 7fe91ac9a640 1 Processor -- start 2024-03-08T08:29:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.879+0000 7fe91ac9a640 1 -- start start 2024-03-08T08:29:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91415ddc0 0x7fe914160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140b9980 con 0x7fe9140d1110 2024-03-08T08:29:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140b8d50 con 0x7fe91415ddc0 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe91ac9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140b7660 con 0x7fe9140d31c0 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57790/0 (socket says 172.21.15.12:57790) 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 -- 172.21.15.12:0/4043628950 learned_addr learned my addr 172.21.15.12:0/4043628950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 -- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 msgr2=0x7fe9140abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe919237640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91415ddc0 0x7fe914160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 -- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91415ddc0 msgr2=0x7fe914160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91415ddc0 0x7fe914160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.880+0000 7fe918a36640 1 -- 172.21.15.12:0/4043628950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140aaa90 con 0x7fe9140d1110 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe918a36640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c460 tx=0x7fe90800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908018070 con 0x7fe9140d1110 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe908007500 con 0x7fe9140d1110 2024-03-08T08:29:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9140aad70 con 0x7fe9140d1110 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908004a20 con 0x7fe9140d1110 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.881+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9140ab330 con 0x7fe9140d1110 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.882+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9140749c0 con 0x7fe9140d1110 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.882+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe90801b070 con 0x7fe9140d1110 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.883+0000 7fe9117fa640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 0x7fe8f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.883+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe908089400 con 0x7fe9140d1110 2024-03-08T08:29:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.883+0000 7fe913fff640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 0x7fe8f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.883+0000 7fe913fff640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 0x7fe8f806d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe900036000 tx=0x7fe900030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:14.887+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe908058310 con 0x7fe9140d1110 2024-03-08T08:29:15.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.069+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9140d0b30 con 0x7fe9140d1110 2024-03-08T08:29:15.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.070+0000 7fe9117fa640 1 -- 172.21.15.12:0/4043628950 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fe908057c10 con 0x7fe9140d1110 2024-03-08T08:29:15.071 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:15.072 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 msgr2=0x7fe8f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 0x7fe8f806d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe900036000 tx=0x7fe900030000 comp rx=0 tx=0).stop 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 msgr2=0x7fe9140aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c460 tx=0x7fe90800c920 comp rx=0 tx=0).stop 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 shutdown_connections 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe8f806b4d0 0x7fe8f806d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91415ddc0 0x7fe914160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9140d31c0 0x7fe9140abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 --2- 172.21.15.12:0/4043628950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9140d1110 0x7fe9140aa4b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 >> 172.21.15.12:0/4043628950 conn(0x7fe914072720 msgr2=0x7fe9140b7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 shutdown_connections 2024-03-08T08:29:15.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.072+0000 7fe91ac9a640 1 -- 172.21.15.12:0/4043628950 wait complete. 2024-03-08T08:29:15.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-08T08:29:15.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-08T08:29:15.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-08T08:29:15.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-08T08:29:15.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-08T08:29:15.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-08T08:29:15.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-08T08:29:15.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-08T08:29:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.256+0000 7f4933f43640 1 Processor -- start 2024-03-08T08:29:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.256+0000 7f4933f43640 1 -- start start 2024-03-08T08:29:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.256+0000 7fa507284640 1 Processor -- start 2024-03-08T08:29:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7fa507284640 1 -- start start 2024-03-08T08:29:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7f4933f43640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f492c15dc30 0x7f492c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7f4933f43640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7f4933f43640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c15e330 con 0x7f492c13fdc0 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7f4933f43640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c15ebd0 con 0x7f492c15dc30 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.257+0000 7f4933f43640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c15e610 con 0x7f492c0cc1f0 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f492c15dc30 0x7f492c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f492c15dc30 0x7f492c0ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57794/0 (socket says 172.21.15.12:57794) 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 -- 172.21.15.12:0/2604780045 learned_addr learned my addr 172.21.15.12:0/2604780045 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f49324e0640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 -- 172.21.15.12:0/2604780045 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 msgr2=0x7f492c0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c0cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 -- 172.21.15.12:0/2604780045 >> v1:172.21.15.156:6789/0 conn(0x7f492c13fdc0 legacy=0x7f492c140250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T08:29:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7f4931cdf640 1 -- 172.21.15.12:0/2604780045 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c15e8f0 con 0x7f492c15dc30 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7fa507284640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5000ceff0 0x7fa5000b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.258+0000 7fa507284640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa5001452d0 0x7fa5000b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa505020640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa5000d08d0 0x7fa5000d0cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47066/0 (socket says 172.21.15.12:47066) 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa50481f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5000ceff0 0x7fa5000b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa50481f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5000ceff0 0x7fa5000b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40316/0 (socket says 172.21.15.12:40316) 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa505020640 1 -- 172.21.15.12:0/3317012363 learned_addr learned my addr 172.21.15.12:0/3317012363 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500071760 con 0x7fa5000d08d0 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa500071a40 con 0x7fa5001452d0 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5001450f0 con 0x7fa5000ceff0 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa505821640 1 --2- 172.21.15.12:0/3317012363 >> v2:172.21.15.156:3300/0 conn(0x7fa5001452d0 0x7fa5000b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903336480 0 0) 0x7fa500071760 con 0x7fa5000d08d0 2024-03-08T08:29:15.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa7c638c640 1 Processor -- start 2024-03-08T08:29:15.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7fa7c638c640 1 -- start start 2024-03-08T08:29:15.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7f4931cdf640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c15dc30 0x7f492c0ce2a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f491c009870 tx=0x7f491c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d515227e68c61f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7f7247fc7640 1 Processor -- start 2024-03-08T08:29:15.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ec003520 con 0x7fa5000d08d0 2024-03-08T08:29:15.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7f4930cdd640 1 -- 172.21.15.12:0/2604780045 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c0317c0 con 0x7f492c15dc30 2024-03-08T08:29:15.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7f4930cdd640 1 -- 172.21.15.12:0/2604780045 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491c037070 con 0x7f492c15dc30 2024-03-08T08:29:15.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7f4930cdd640 1 -- 172.21.15.12:0/2604780045 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c031b80 con 0x7f492c15dc30 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7f8eb1356640 1 Processor -- start 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749015373 0 0) 0x7fa4ec003520 con 0x7fa5000d08d0 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5000ceff0 msgr2=0x7fa5000b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5000ceff0 0x7fa5000b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 >> v2:172.21.15.156:3300/0 conn(0x7fa5001452d0 msgr2=0x7fa5000b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 --2- 172.21.15.12:0/3317012363 >> v2:172.21.15.156:3300/0 conn(0x7fa5001452d0 0x7fa5000b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5000b47e0 con 0x7fa5000d08d0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fa4fc001670 con 0x7fa5000d08d0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 >> v1:172.21.15.156:6789/0 conn(0x7fa5000d08d0 legacy=0x7fa5000d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa4ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0051c0 0x7fa4ec00fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa500071a40 con 0x7fa4ec004800 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5001450f0 con 0x7fa4ec0039e0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec010410 con 0x7fa4ec0051c0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa505020640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa4ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa505821640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0051c0 0x7fa4ec00fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa50481f640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7f7247fc7640 1 -- start start 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 -- 172.21.15.12:0/2604780045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c15dc30 msgr2=0x7f492c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c15dc30 0x7f492c0ce2a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f491c009870 tx=0x7f491c02b860 comp rx=0 tx=0).stop 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 -- 172.21.15.12:0/2604780045 shutdown_connections 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 --2- 172.21.15.12:0/2604780045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c15dc30 0x7f492c0ce2a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 -- 172.21.15.12:0/2604780045 >> 172.21.15.12:0/2604780045 conn(0x7f492c06abd0 msgr2=0x7f492c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 -- 172.21.15.12:0/2604780045 shutdown_connections 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.259+0000 7f8eb1356640 1 -- start start 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f8eb1356640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8eac161fe0 0x7f8eac0af010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa7c638c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c016cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa7c638c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c006f0e0 con 0x7fa7c0162f30 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa7c638c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c006f350 con 0x7fa7c0162b50 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.260+0000 7fa7c638c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006fc80 con 0x7fa7c00b4020 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7c4929640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c016cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7c4929640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c016cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40328/0 (socket says 172.21.15.12:40328) 2024-03-08T08:29:15.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7c4929640 1 -- 172.21.15.12:0/2735235478 learned_addr learned my addr 172.21.15.12:0/2735235478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805545236 0 0) 0x7fa7c006f0e0 con 0x7fa7c0162f30 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003520 con 0x7fa7c0162f30 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f8eb1356640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac06d540 0x7f8eac0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f8eb1356640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac1623c0 con 0x7f8eac071a20 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f8eb1356640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac071e70 con 0x7f8eac161fe0 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f8eb1356640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac06da20 con 0x7f8eac06d540 2024-03-08T08:29:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4153302643 0 0) 0x7fa7a8003520 con 0x7fa7c0162f30 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 msgr2=0x7fa7c016cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c016cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 >> v1:172.21.15.156:6789/0 conn(0x7fa7c0162b50 legacy=0x7fa7c0066ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c006f630 con 0x7fa7c0162f30 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f7247fc7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f724015fca0 0x7f72400b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa505020640 1 -- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0051c0 msgr2=0x7fa4ec00fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa505020640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0051c0 0x7fa4ec00fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa505020640 1 -- 172.21.15.12:0/3317012363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4ec004800 msgr2=0x7fa4ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa505020640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7fa505020640 1 -- 172.21.15.12:0/3317012363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5000b47e0 con 0x7fa4ec0039e0 2024-03-08T08:29:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f7247fc7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7240072470 0x7f7240071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7247fc7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7240163470 con 0x7f7240071b50 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7247fc7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400b3120 con 0x7f7240072470 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7247fc7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400cf850 con 0x7f724015fca0 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7246564640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7240071b50 0x7f7240165a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47088/0 (socket says 172.21.15.12:47088) 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fa7b00019f0 con 0x7fa7c0162f30 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 >> v1:172.21.15.12:6789/0 conn(0x7fa7c0162f30 legacy=0x7fa7c00b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7a80039e0 0x7fa7a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7246564640 1 -- 172.21.15.12:0/4053781666 learned_addr learned my addr 172.21.15.12:0/4053781666 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.261+0000 7f4933f43640 1 -- 172.21.15.12:0/2604780045 wait complete. 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a8004800 0x7fa7a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006fc80 con 0x7fa7a8004800 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a800bad0 con 0x7fa7a80039e0 2024-03-08T08:29:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a800bcd0 con 0x7fa7c00b4020 2024-03-08T08:29:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7bffff640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7a80039e0 0x7fa7a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7245d63640 1 --2- 172.21.15.12:0/4053781666 >> v2:172.21.15.156:3300/0 conn(0x7f724015fca0 0x7f72400b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa50481f640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f7245562640 1 --2- 172.21.15.12:0/4053781666 >> v2:172.21.15.12:3300/0 conn(0x7f7240072470 0x7f7240071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7bf7fe640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a8004800 0x7fa7a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f80920b7640 1 Processor -- start 2024-03-08T08:29:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f80920b7640 1 -- start start 2024-03-08T08:29:15.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa505020640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa4ec003e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00c240 tx=0x7fa4ec005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59df7a6c56404ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc007d30 con 0x7fa4ec0039e0 2024-03-08T08:29:15.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc030070 con 0x7fa4ec0039e0 2024-03-08T08:29:15.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa4f7fff640 1 -- 172.21.15.12:0/3317012363 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc02cd90 con 0x7fa4ec0039e0 2024-03-08T08:29:15.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f7245562640 1 -- 172.21.15.12:0/4053781666 >> v2:172.21.15.156:3300/0 conn(0x7f724015fca0 msgr2=0x7f72400b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f4933f43640 1 Processor -- start 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f4933f43640 1 -- start start 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8eac161fe0 0x7f8eac0af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8eac161fe0 0x7f8eac0af010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53288/0 (socket says 172.21.15.12:53288) 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 -- 172.21.15.12:0/3815859617 learned_addr learned my addr 172.21.15.12:0/3815859617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaa7fc640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac06d540 0x7f8eac0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 -- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac06d540 msgr2=0x7f8eac0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac06d540 0x7f8eac0714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 -- 172.21.15.12:0/3815859617 >> v1:172.21.15.12:6789/0 conn(0x7f8eac071a20 legacy=0x7f8eac0b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.262+0000 7f8eaaffd640 1 -- 172.21.15.12:0/3815859617 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac06dd00 con 0x7f8eac161fe0 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7c4929640 1 -- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7a80039e0 msgr2=0x7fa7a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7a80039e0 0x7fa7a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7c4929640 1 -- 172.21.15.12:0/2735235478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a8004800 msgr2=0x7fa7a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a8004800 0x7fa7a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7fa7c4929640 1 -- 172.21.15.12:0/2735235478 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8003520 con 0x7fa7c00b4020 2024-03-08T08:29:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8084048040 0x7f808404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac161fe0 0x7f8eac0af010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0723a0 tx=0x7f8e9c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97dfcb50ebbd9fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f8084038010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8084038550 0x7f80840389e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8084048420 con 0x7f8084048040 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80840547a0 con 0x7f8084038550 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f80920b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8084053700 con 0x7f8084037bb0 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8084048040 0x7f808404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f7245562640 1 --2- 172.21.15.12:0/4053781666 >> v2:172.21.15.156:3300/0 conn(0x7f724015fca0 0x7f72400b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f7245562640 1 -- 172.21.15.12:0/4053781666 >> v1:172.21.15.156:6789/0 conn(0x7f7240071b50 legacy=0x7f7240165a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.263+0000 7f7245562640 1 -- 172.21.15.12:0/4053781666 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240071f60 con 0x7f7240072470 2024-03-08T08:29:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f9f159c7640 1 Processor -- start 2024-03-08T08:29:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8084048040 0x7f808404fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57834/0 (socket says 172.21.15.12:57834) 2024-03-08T08:29:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 -- 172.21.15.12:0/3006689251 learned_addr learned my addr 172.21.15.12:0/3006689251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80908b4640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f8084038010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f4933f43640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f9f159c7640 1 -- start start 2024-03-08T08:29:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80918b6640 1 --2- 172.21.15.12:0/3006689251 >> v2:172.21.15.156:3300/0 conn(0x7f8084038550 0x7f80840389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 msgr2=0x7fa4ec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7fa507284640 1 --2- 172.21.15.12:0/3317012363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa4ec003e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00c240 tx=0x7fa4ec005830 comp rx=0 tx=0).stop 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 -- 172.21.15.12:0/3006689251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 msgr2=0x7f8084038010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f8084038010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 -- 172.21.15.12:0/3006689251 >> v2:172.21.15.156:3300/0 conn(0x7f8084038550 msgr2=0x7f80840389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 --2- 172.21.15.12:0/3006689251 >> v2:172.21.15.156:3300/0 conn(0x7f8084038550 0x7f80840389e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f80910b5640 1 -- 172.21.15.12:0/3006689251 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840539e0 con 0x7f8084048040 2024-03-08T08:29:15.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f4931cdf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7fa505821640 1 -- 172.21.15.12:0/3317012363 reap_dead start 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f4933f43640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c13fdc0 0x7f492c1590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4933f43640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f492c15dc30 0x7f492c14f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4933f43640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0ce7e0 con 0x7f492c13fdc0 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4933f43640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c15e010 con 0x7f492c15dc30 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4933f43640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0ce960 con 0x7f492c0cc1f0 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4931cdf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c1599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40356/0 (socket says 172.21.15.12:40356) 2024-03-08T08:29:15.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f4931cdf640 1 -- 172.21.15.12:0/3373575779 learned_addr learned my addr 172.21.15.12:0/3373575779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f8ea9ffb640 1 -- 172.21.15.12:0/3815859617 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c035020 con 0x7f8eac161fe0 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f8ea9ffb640 1 -- 172.21.15.12:0/3815859617 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c031640 con 0x7f8eac161fe0 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 -- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac161fe0 msgr2=0x7f8eac0af010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac161fe0 0x7f8eac0af010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0723a0 tx=0x7f8e9c02b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 -- 172.21.15.12:0/3815859617 shutdown_connections 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac06d540 0x7f8eac0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 --2- 172.21.15.12:0/3815859617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac161fe0 0x7f8eac0af010 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 -- 172.21.15.12:0/3815859617 >> 172.21.15.12:0/3815859617 conn(0x7f8eac06abd0 msgr2=0x7f8eac06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f49324e0640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f492c15dc30 0x7f492c14f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7a80051c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00732b0 tx=0x7fa7a8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17006ac3dcc181e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4039070 con 0x7fa7c00b4020 2024-03-08T08:29:15.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b402c780 con 0x7fa7c00b4020 2024-03-08T08:29:15.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.264+0000 7f7245562640 1 --2- 172.21.15.12:0/4053781666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f7240071610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f724006c9d0 tx=0x7f723802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286500ce5c9c5247 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f49314de640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c13fdc0 0x7f492c1590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f7244d61640 1 -- 172.21.15.12:0/4053781666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238039070 con 0x7f7240072470 2024-03-08T08:29:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f80910b5640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084048040 0x7f808404fa90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f808400ada0 tx=0x7f808802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70bd926598ef61d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f7244d61640 1 -- 172.21.15.12:0/4053781666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723802cc00 con 0x7f7240072470 2024-03-08T08:29:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f159c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9f08055740 0x7f9f08034ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0effd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9f080e00d0 0x7f9f08038e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47100/0 (socket says 172.21.15.12:47100) 2024-03-08T08:29:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 -- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 msgr2=0x7f492c1599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f492c0cc1f0 0x7f492c1599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 -- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c13fdc0 msgr2=0x7f492c1590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f492c13fdc0 0x7f492c1590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c143400 con 0x7f492c15dc30 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f8083fff640 1 -- 172.21.15.12:0/3006689251 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088034070 con 0x7f8084048040 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f8083fff640 1 -- 172.21.15.12:0/3006689251 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8088030860 con 0x7f8084048040 2024-03-08T08:29:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 -- 172.21.15.12:0/4053781666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 msgr2=0x7f7240071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 --2- 172.21.15.12:0/4053781666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f7240071610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f724006c9d0 tx=0x7f723802be50 comp rx=0 tx=0).stop 2024-03-08T08:29:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f49324e0640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f492c15dc30 0x7f492c14f360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f492800d530 tx=0x7f492800d9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 -- 172.21.15.12:0/4053781666 shutdown_connections 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 --2- 172.21.15.12:0/4053781666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f7240071610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 --2- 172.21.15.12:0/4053781666 >> v2:172.21.15.156:3300/0 conn(0x7f724015fca0 0x7f72400b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7fa7beffd640 1 -- 172.21.15.12:0/2735235478 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40097e0 con 0x7fa7c00b4020 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f7247fc7640 1 -- 172.21.15.12:0/4053781666 >> 172.21.15.12:0/4053781666 conn(0x7f72400cf9f0 msgr2=0x7f72400cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7fcf8c9c0640 1 Processor -- start 2024-03-08T08:29:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 shutdown_connections 2024-03-08T08:29:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 >> 172.21.15.12:0/3317012363 conn(0x7fa50006f480 msgr2=0x7fa50006f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fcf8c9c0640 1 -- start start 2024-03-08T08:29:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49280125f0 con 0x7f492c15dc30 2024-03-08T08:29:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 -- 172.21.15.12:0/3815859617 shutdown_connections 2024-03-08T08:29:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.265+0000 7f8eb1356640 1 -- 172.21.15.12:0/3815859617 wait complete. 2024-03-08T08:29:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f4933f43640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c1436e0 con 0x7f492c15dc30 2024-03-08T08:29:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 shutdown_connections 2024-03-08T08:29:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4928019070 con 0x7f492c15dc30 2024-03-08T08:29:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa7c638c640 1 -- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 msgr2=0x7fa7a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa7c638c640 1 --2- 172.21.15.12:0/2735235478 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7a80051c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00732b0 tx=0x7fa7a8005830 comp rx=0 tx=0).stop 2024-03-08T08:29:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa7c4929640 1 -- 172.21.15.12:0/2735235478 reap_dead start 2024-03-08T08:29:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49280129b0 con 0x7f492c15dc30 2024-03-08T08:29:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa7c638c640 1 -- 172.21.15.12:0/2735235478 shutdown_connections 2024-03-08T08:29:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa7c638c640 1 -- 172.21.15.12:0/2735235478 >> 172.21.15.12:0/2735235478 conn(0x7fa7c006db00 msgr2=0x7fa7c006dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 -- 172.21.15.12:0/3006689251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084048040 msgr2=0x7f808404fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084048040 0x7f808404fa90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f808400ada0 tx=0x7f808802b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7fa507284640 1 -- 172.21.15.12:0/3317012363 wait complete. 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 -- 172.21.15.12:0/3006689251 shutdown_connections 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 --2- 172.21.15.12:0/3006689251 >> v2:172.21.15.156:3300/0 conn(0x7f8084038550 0x7f80840389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f8084038010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 --2- 172.21.15.12:0/3006689251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084048040 0x7f808404fa90 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f808400ada0 tx=0x7f808802b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f80920b7640 1 -- 172.21.15.12:0/3006689251 >> 172.21.15.12:0/3006689251 conn(0x7f8084009fb0 msgr2=0x7f808400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0effd640 1 -- 172.21.15.12:0/1269497460 learned_addr learned my addr 172.21.15.12:0/1269497460 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f08034d40 con 0x7f9f08050460 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f08007d60 con 0x7f9f080e00d0 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f08050960 con 0x7f9f08055740 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0f7fe640 1 --2- 172.21.15.12:0/1269497460 >> v2:172.21.15.156:3300/0 conn(0x7f9f08055740 0x7f9f08034ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0dffb640 1 -- 172.21.15.12:0/1269497460 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955046093 0 0) 0x7f9f08007d60 con 0x7f9f080e00d0 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0dffb640 1 -- 172.21.15.12:0/1269497460 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef800bd30 con 0x7f9f080e00d0 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/1269497460 >> v1:172.21.15.156:6789/0 conn(0x7f9f080e00d0 legacy=0x7f9f08038e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/1269497460 >> v1:172.21.15.12:6789/0 conn(0x7f9f08050460 legacy=0x7f9f08055030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.266+0000 7f9f0e7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9f08050460 0x7f9f08055030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52790/0 (socket says 172.21.15.12:52790) 2024-03-08T08:29:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f4933f43640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c16bea0 con 0x7f492c15dc30 2024-03-08T08:29:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/1269497460 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08008040 con 0x7f9f08055740 2024-03-08T08:29:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf8c9c0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf880ce980 0x7fcf880b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf8c9c0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf8c9c0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880ce5a0 con 0x7fcf8815dc80 2024-03-08T08:29:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf8c9c0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf880cf0f0 con 0x7fcf880ce980 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf8c9c0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880cee10 con 0x7fcf880b5780 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40366/0 (socket says 172.21.15.12:40366) 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 -- 172.21.15.12:0/3905138226 learned_addr learned my addr 172.21.15.12:0/3905138226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf86ffd640 1 --2- 172.21.15.12:0/3905138226 >> v2:172.21.15.12:3300/0 conn(0x7fcf880ce980 0x7fcf880b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f9f0f7fe640 1 --2- 172.21.15.12:0/1269497460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08055740 0x7f9f08034ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000dbc0 tx=0x7f9f0002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e4e525af6ca18b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f8eb1356640 1 Processor -- start 2024-03-08T08:29:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.267+0000 7f8eb1356640 1 -- start start 2024-03-08T08:29:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8eac06d540 0x7f8eac159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac071a20 0x7f8eac1ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac157820 0x7f8eac157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac06e5a0 con 0x7f8eac06d540 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac06e2c0 con 0x7f8eac071a20 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f8eb1356640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0aece0 con 0x7f8eac157820 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f8eab7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac157820 0x7f8eac157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f8eab7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac157820 0x7f8eac157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40376/0 (socket says 172.21.15.12:40376) 2024-03-08T08:29:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f8eab7fe640 1 -- 172.21.15.12:0/616407753 learned_addr learned my addr 172.21.15.12:0/616407753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7fa7c638c640 1 -- 172.21.15.12:0/2735235478 shutdown_connections 2024-03-08T08:29:15.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f80920b7640 1 -- 172.21.15.12:0/3006689251 shutdown_connections 2024-03-08T08:29:15.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f9f0dffb640 1 -- 172.21.15.12:0/1269497460 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000b450 con 0x7f9f08055740 2024-03-08T08:29:15.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f9f0dffb640 1 -- 172.21.15.12:0/1269497460 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000b5f0 con 0x7f9f08055740 2024-03-08T08:29:15.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f9f0dffb640 1 -- 172.21.15.12:0/1269497460 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000b9c0 con 0x7f9f08055740 2024-03-08T08:29:15.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7fa507284640 1 Processor -- start 2024-03-08T08:29:15.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4928012b50 con 0x7f492c15dc30 2024-03-08T08:29:15.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7fa507284640 1 -- start start 2024-03-08T08:29:15.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8eac06d540 0x7f8eac159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 -- 172.21.15.12:0/3905138226 >> v2:172.21.15.12:3300/0 conn(0x7fcf880ce980 msgr2=0x7fcf880b6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 --2- 172.21.15.12:0/3905138226 >> v2:172.21.15.12:3300/0 conn(0x7fcf880ce980 0x7fcf880b6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 -- 172.21.15.12:0/3905138226 >> v1:172.21.15.12:6789/0 conn(0x7fcf8815dc80 legacy=0x7fcf88162230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7fcf87fff640 1 -- 172.21.15.12:0/3905138226 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880d1ec0 con 0x7fcf880b5780 2024-03-08T08:29:15.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7fa7c638c640 1 -- 172.21.15.12:0/2735235478 wait complete. 2024-03-08T08:29:15.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaa7fc640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac071a20 0x7f8eac1ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f4933f43640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f492c0749c0 con 0x7f492c15dc30 2024-03-08T08:29:15.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7fcf87fff640 1 --2- 172.21.15.12:0/3905138226 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009e50 tx=0x7fcf8002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=926b1c5322770ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08055740 msgr2=0x7f9f08034ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f9f159c7640 1 --2- 172.21.15.12:0/1269497460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08055740 0x7f9f08034ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000dbc0 tx=0x7f9f0002b840 comp rx=0 tx=0).stop 2024-03-08T08:29:15.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7fcf867fc640 1 -- 172.21.15.12:0/3905138226 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8002cb90 con 0x7fcf880b5780 2024-03-08T08:29:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7fcf867fc640 1 -- 172.21.15.12:0/3905138226 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8002cd30 con 0x7fcf880b5780 2024-03-08T08:29:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7fcf867fc640 1 -- 172.21.15.12:0/3905138226 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80034610 con 0x7fcf880b5780 2024-03-08T08:29:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f49227fc640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f491006b5a0 0x7f491006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 -- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac157820 msgr2=0x7f8eac157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eac157820 0x7f8eac157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 -- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac071a20 msgr2=0x7f8eac1ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac071a20 0x7f8eac1ffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac162630 con 0x7f8eac06d540 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaa7fc640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eac071a20 0x7f8eac1ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f4931cdf640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f491006b5a0 0x7f491006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 shutdown_connections 2024-03-08T08:29:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f9f159c7640 1 --2- 172.21.15.12:0/1269497460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08055740 0x7f9f08034ff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7fa507284640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa50015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 >> 172.21.15.12:0/1269497460 conn(0x7f9f08055c00 msgr2=0x7f9f08056010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f492808a480 con 0x7f492c15dc30 2024-03-08T08:29:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa507284640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa50013ef90 0x7fa500159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f7247fc7640 1 -- 172.21.15.12:0/4053781666 shutdown_connections 2024-03-08T08:29:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.268+0000 7f7247fc7640 1 -- 172.21.15.12:0/4053781666 wait complete. 2024-03-08T08:29:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f7247fc7640 1 Processor -- start 2024-03-08T08:29:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f7247fc7640 1 -- start start 2024-03-08T08:29:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa505020640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa50015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa505020640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa50015da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53368/0 (socket says 172.21.15.12:53368) 2024-03-08T08:29:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f49227fc640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f49100737c0 0x7f4910075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.270+0000 7f8eaaffd640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8eac06d540 0x7f8eac159ab0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c033000 tx=0x7f8e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa505020640 1 -- 172.21.15.12:0/2629982285 learned_addr learned my addr 172.21.15.12:0/2629982285 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f7247fc7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7240071b50 0x7f72400ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f7247fc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f72400cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4928022ec0 con 0x7f49100737c0 2024-03-08T08:29:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa507284640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50015c090 0x7fa50015e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4928091050 con 0x7f492c15dc30 2024-03-08T08:29:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.269+0000 7f80920b7640 1 -- 172.21.15.12:0/3006689251 wait complete. 2024-03-08T08:29:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa505821640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50015c090 0x7fa50015e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa500144a20 con 0x7fa50015c090 2024-03-08T08:29:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f49314de640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f49100737c0 0x7f4910075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa500144d80 con 0x7fa4ec0039e0 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa500144ba0 con 0x7fa50013ef90 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/3905138226 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 msgr2=0x7fcf880b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/3905138226 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009e50 tx=0x7fcf8002bff0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/3905138226 shutdown_connections 2024-03-08T08:29:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f80920b7640 1 Processor -- start 2024-03-08T08:29:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/3905138226 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b5780 0x7fcf880b5c10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/3905138226 >> v2:172.21.15.12:3300/0 conn(0x7fcf880ce980 0x7fcf880b6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/3905138226 >> 172.21.15.12:0/3905138226 conn(0x7fcf880d07c0 msgr2=0x7fcf880d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 -- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa50013ef90 msgr2=0x7fa500159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa50481f640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa50013ef90 0x7fa500159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/3905138226 shutdown_connections 2024-03-08T08:29:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7f7245562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f72400cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/3905138226 wait complete. 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa50013ef90 0x7fa500159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 -- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50015c090 msgr2=0x7fa50015e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50015c090 0x7fa50015e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50015c990 con 0x7fa4ec0039e0 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505821640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50015c090 0x7fa50015e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f49314de640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f49100737c0 0x7f4910075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7247fc7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f724015fca0 0x7f7240161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7247fc7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7240162c70 con 0x7f7240072470 2024-03-08T08:29:15.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f4931cdf640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f491006b5a0 0x7f491006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f491c0090b0 tx=0x7f491c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7245562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f72400cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57868/0 (socket says 172.21.15.12:57868) 2024-03-08T08:29:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7245562640 1 -- 172.21.15.12:0/3742301801 learned_addr learned my addr 172.21.15.12:0/3742301801 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 shutdown_connections 2024-03-08T08:29:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7245d63640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7240071b50 0x7f72400ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7240162990 con 0x7f7240071b50 2024-03-08T08:29:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c040070 con 0x7f8eac06d540 2024-03-08T08:29:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c039b70 con 0x7f8eac06d540 2024-03-08T08:29:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c035040 con 0x7f8eac06d540 2024-03-08T08:29:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa505020640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa50015da30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0017e0 tx=0x7fa4fc001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f9f159c7640 1 -- 172.21.15.12:0/1269497460 wait complete. 2024-03-08T08:29:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa7c638c640 1 Processor -- start 2024-03-08T08:29:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.271+0000 7fa7c638c640 1 -- start start 2024-03-08T08:29:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc004d50 con 0x7fa4ec0039e0 2024-03-08T08:29:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc007070 con 0x7fa4ec0039e0 2024-03-08T08:29:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== osd.3 v2:172.21.15.12:6810/2394545747 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4928022ec0 con 0x7f49100737c0 2024-03-08T08:29:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f8eb1356640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac1628a0 con 0x7f8eac06d540 2024-03-08T08:29:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f8eb1356640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac162e60 con 0x7f8eac06d540 2024-03-08T08:29:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c0159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c00ce700 0x7fa7c0161af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7c0162030 0x7fa7c0157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0070110 con 0x7fa7c00ce700 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01633e0 con 0x7fa7c0162030 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7bffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c0159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7bffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c0159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40412/0 (socket says 172.21.15.12:40412) 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7bffff640 1 -- 172.21.15.12:0/2361896558 learned_addr learned my addr 172.21.15.12:0/2361896558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7c638c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c015ddb0 con 0x7fa7c00b4020 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7fa7bf7fe640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c00ce700 0x7fa7c0161af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7c0162030 0x7fa7c0157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bf7fe640 1 -- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 msgr2=0x7fa7c0159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bf7fe640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c0159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bf7fe640 1 -- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7c0162030 msgr2=0x7fa7c0157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bf7fe640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7c0162030 0x7fa7c0157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bf7fe640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c015a1f0 con 0x7fa7c00ce700 2024-03-08T08:29:15.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa7bffff640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7c00b4020 0x7fa7c0159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.272+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7240162f50 con 0x7f724015fca0 2024-03-08T08:29:15.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7f7246564640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f724015fca0 0x7f7240161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc007460 con 0x7fa4ec0039e0 2024-03-08T08:29:15.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50015cc70 con 0x7fa4ec0039e0 2024-03-08T08:29:15.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7fcf8c9c0640 1 Processor -- start 2024-03-08T08:29:15.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.274+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50015d230 con 0x7fa4ec0039e0 2024-03-08T08:29:15.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.274+0000 7fcf8c9c0640 1 -- start start 2024-03-08T08:29:15.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.273+0000 7f80920b7640 1 -- start start 2024-03-08T08:29:15.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.274+0000 7f8eb1356640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8eac163230 con 0x7f8eac06d540 2024-03-08T08:29:15.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.274+0000 7f9f159c7640 1 Processor -- start 2024-03-08T08:29:15.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.274+0000 7fa7bf7fe640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c00ce700 0x7fa7c0161af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0002230 tx=0x7fa7b000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f808404dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084038550 0x7f808404a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80840e59b0 0x7f80841894b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084054280 con 0x7f8084038550 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084053fa0 con 0x7f80840e59b0 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80920b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808404ffd0 con 0x7f8084037bb0 2024-03-08T08:29:15.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f9f159c7640 1 -- start start 2024-03-08T08:29:15.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf880b5780 0x7fcf881fa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa500071760 con 0x7fa4ec0039e0 2024-03-08T08:29:15.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80918b6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80840e59b0 0x7f80841894b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f808404dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 0x7fcf880b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f808404dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40418/0 (socket says 172.21.15.12:40418) 2024-03-08T08:29:15.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 -- 172.21.15.12:0/4193951548 learned_addr learned my addr 172.21.15.12:0/4193951548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b8110 0x7fcf881f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8815e060 con 0x7fcf8815dc80 2024-03-08T08:29:15.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880d2480 con 0x7fcf880b5780 2024-03-08T08:29:15.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80908b4640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084038550 0x7f808404a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf86ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 0x7fcf880b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf86ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 0x7fcf880b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57890/0 (socket says 172.21.15.12:57890) 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf86ffd640 1 -- 172.21.15.12:0/442498718 learned_addr learned my addr 172.21.15.12:0/442498718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf8c9c0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881fbcf0 con 0x7fcf880b8110 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf877fe640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf880b5780 0x7fcf881fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 -- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80840e59b0 msgr2=0x7f80841894b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80840e59b0 0x7f80841894b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 -- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084038550 msgr2=0x7f808404a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8084038550 0x7f808404a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7f80910b5640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808404af60 con 0x7f8084037bb0 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.275+0000 7fcf87fff640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b8110 0x7fcf881f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f7246564640 1 -- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7240071b50 msgr2=0x7f72400ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f7246564640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7240071b50 0x7f72400ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f7246564640 1 -- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 msgr2=0x7f72400cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f7246564640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7240072470 0x7f72400cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f7246564640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400cec40 con 0x7f724015fca0 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f80910b5640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f808404dad0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f808802bf30 tx=0x7f8088004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088048070 con 0x7f8084037bb0 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8088030660 con 0x7f8084037bb0 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088034040 con 0x7f8084037bb0 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7fcf86ffd640 1 -- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b8110 msgr2=0x7fcf881f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7fcf86ffd640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf880b8110 0x7fcf881f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808404b240 con 0x7f8084037bb0 2024-03-08T08:29:15.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8084188fb0 con 0x7f8084037bb0 2024-03-08T08:29:15.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08050460 0x7f9f080d1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f08055740 0x7f9f08037170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 0x7f9f0802d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f0f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 0x7f9f0802d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f0f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 0x7f9f0802d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57896/0 (socket says 172.21.15.12:57896) 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f0e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f08055740 0x7f9f08037170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/2195287374 learned_addr learned my addr 172.21.15.12:0/2195287374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f08039520 con 0x7f9f080e00d0 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f08008600 con 0x7f9f08050460 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f08008b20 con 0x7f9f08055740 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.276+0000 7f9f0effd640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08050460 0x7f9f080d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f08055740 msgr2=0x7f9f08037170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f08055740 0x7f9f08037170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08050460 msgr2=0x7f9f080d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f08050460 0x7f9f080d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4fc007600 con 0x7fa4ec0039e0 2024-03-08T08:29:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080dee90 con 0x7f9f080e00d0 2024-03-08T08:29:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f7246564640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f724015fca0 0x7f7240161400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f723000c390 tx=0x7f723000c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fcf86ffd640 1 -- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf880b5780 msgr2=0x7fcf881fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fcf86ffd640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf880b5780 0x7fcf881fa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7f9f0f7fe640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 0x7f9f0802d1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000dbc0 tx=0x7f9f000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fcf86ffd640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf881fabd0 con 0x7fcf8815dc80 2024-03-08T08:29:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b000fe40 con 0x7fa7c00ce700 2024-03-08T08:29:15.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa4e406b500 0x7fa4e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4fc0abf80 con 0x7fa4ec0039e0 2024-03-08T08:29:15.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7fa4e4073700 0x7fa4e4075b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4fc03ee50 con 0x7fa4e4073700 2024-03-08T08:29:15.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4fc0ac330 con 0x7fa4ec0039e0 2024-03-08T08:29:15.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7fa50481f640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa4e406b500 0x7fa4e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7230017070 con 0x7f724015fca0 2024-03-08T08:29:15.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8084005390 con 0x7f8084037bb0 2024-03-08T08:29:15.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230005ce0 con 0x7f724015fca0 2024-03-08T08:29:15.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400c5960 con 0x7f724015fca0 2024-03-08T08:29:15.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000b450 con 0x7f9f080e00d0 2024-03-08T08:29:15.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f00033bc0 con 0x7f9f080e00d0 2024-03-08T08:29:15.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00006e10 con 0x7f9f080e00d0 2024-03-08T08:29:15.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7fa505821640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7fa4e4073700 0x7fa4e4075b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7fa50481f640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa4e406b500 0x7fa4e406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa50015e250 tx=0x7fa4f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.277+0000 7fcf877fe640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf880b5780 0x7fcf881fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72400ccde0 con 0x7f724015fca0 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72300072f0 con 0x7f724015fca0 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7fa7c638c640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c00ce410 con 0x7fa7c00ce700 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7fa7c638c640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c0200810 con 0x7fa7c00ce700 2024-03-08T08:29:15.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b0018070 con 0x7fa7c00ce700 2024-03-08T08:29:15.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0003970 con 0x7fa7c00ce700 2024-03-08T08:29:15.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8088037070 con 0x7f8084037bb0 2024-03-08T08:29:15.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7fcf86ffd640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 0x7fcf880b7bd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcf7400d830 tx=0x7fcf7400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7f8081ffb640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f807006b4d0 0x7f807006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f080d1590 con 0x7f9f080e00d0 2024-03-08T08:29:15.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.278+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08037e80 con 0x7f9f080e00d0 2024-03-08T08:29:15.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e9c04b060 con 0x7f8eac06d540 2024-03-08T08:29:15.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7fa505821640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7fa4e4073700 0x7fa4e4075b20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8088077000 con 0x7f8084037bb0 2024-03-08T08:29:15.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.279+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7c00749c0 con 0x7fa7c00ce700 2024-03-08T08:29:15.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7400b6c0 con 0x7fcf8815dc80 2024-03-08T08:29:15.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f08005810 con 0x7f9f080e00d0 2024-03-08T08:29:15.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f8081ffb640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7f8070073670 0x7f8070075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8088041e50 con 0x7f8070073670 2024-03-08T08:29:15.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f80908b4640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f807006b4d0 0x7f807006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.280+0000 7f80918b6640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7f8070073670 0x7f8070075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf881fb170 con 0x7fcf8815dc80 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881fb730 con 0x7fcf8815dc80 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf74019070 con 0x7fcf8815dc80 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7400b6c0 con 0x7fcf8815dc80 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7228001030 con 0x7f724015fca0 2024-03-08T08:29:15.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f723001a070 con 0x7f724015fca0 2024-03-08T08:29:15.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7f80918b6640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7f8070073670 0x7f8070075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.281+0000 7f80908b4640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f807006b4d0 0x7f807006d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f807c009a00 tx=0x7f807c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== osd.0 v2:172.21.15.12:6809/1155150761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4fc03ee50 con 0x7fa4e4073700 2024-03-08T08:29:15.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8e937fe640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8e9806b5a0 0x7f8e9806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e9c0abfa0 con 0x7f8eac06d540 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8e937fe640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7f8e980737c0 0x7f8e98075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e9c033ec0 con 0x7f8e980737c0 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e9c0ac390 con 0x7f8eac06d540 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f72367fc640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f722406b5a0 0x7f722406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7230089460 con 0x7f724015fca0 2024-03-08T08:29:15.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f72367fc640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f7224073740 0x7f7224075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== osd.7 v2:172.21.15.156:6808/1139145947 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8088041e50 con 0x7f8070073670 2024-03-08T08:29:15.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8eab7fe640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7f8e980737c0 0x7f8e98075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f8eaa7fc640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8e9806b5a0 0x7f8e9806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7230015e70 con 0x7f7224073740 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7f7245562640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f7224073740 0x7f7224075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7f7245d63640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f722406b5a0 0x7f722406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7f7245562640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f7224073740 0x7f7224075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7f7245d63640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f722406b5a0 0x7f722406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f723c009aa0 tx=0x7f723c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7f8eaa7fc640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8e9806b5a0 0x7f8e9806da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06ebf0 tx=0x7f8ea0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f00044030 con 0x7f9f080e00d0 2024-03-08T08:29:15.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9ef7fff640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9efc06b5a0 0x7f9efc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f0002f020 con 0x7f9f080e00d0 2024-03-08T08:29:15.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9ef7fff640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f9efc0737c0 0x7f9efc075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0000b5f0 con 0x7f9efc0737c0 2024-03-08T08:29:15.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f8eab7fe640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7f8e980737c0 0x7f8e98075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f000ac6c0 con 0x7f9f080e00d0 2024-03-08T08:29:15.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf64001030 con 0x7fcf8815dc80 2024-03-08T08:29:15.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9f0effd640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9efc06b5a0 0x7f9efc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7f9f0e7fc640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f9efc0737c0 0x7f9efc075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.282+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7b0003b10 con 0x7fa7c00ce700 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bcff9640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa7ac06b500 0x7fa7ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7b008b1b0 con 0x7fa7c00ce700 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bcff9640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7fa7ac073720 0x7fa7ac075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7b001ee50 con 0x7fa7ac073720 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7b008b5a0 con 0x7fa7c00ce700 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.283+0000 7fa7bffff640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa7ac06b500 0x7fa7ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7fa7ac073720 0x7fa7ac075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.284+0000 7fa7bffff640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa7ac06b500 0x7fa7ac06d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8001660 tx=0x7fa7b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7fa7c4929640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7fa7ac073720 0x7fa7ac075b40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7f9f0effd640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9efc06b5a0 0x7f9efc06da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f1004fd70 tx=0x7f9f1006e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7f9f0e7fc640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f9efc0737c0 0x7f9efc075be0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== osd.2 v2:172.21.15.12:6800/2701699187 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7230015e70 con 0x7f7224073740 2024-03-08T08:29:15.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8088076c30 con 0x7f8084037bb0 2024-03-08T08:29:15.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.285+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== osd.4 v2:172.21.15.156:6816/602210282 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8e9c033ec0 con 0x7f8e980737c0 2024-03-08T08:29:15.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.286+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7230090050 con 0x7f724015fca0 2024-03-08T08:29:15.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.286+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== osd.6 v2:172.21.15.156:6800/2822268422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa7b001ee50 con 0x7fa7ac073720 2024-03-08T08:29:15.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.286+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== osd.5 v2:172.21.15.156:6824/1688693160 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9f0000b5f0 con 0x7f9efc0737c0 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.288+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf7400ba70 con 0x7fcf8815dc80 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf7ffff640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf7006b8d0 0x7fcf7006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf7400f040 con 0x7fcf8815dc80 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf7ffff640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7fcf70073a70 0x7fcf70075e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf70076540 con 0x7fcf70073a70 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf7408ae80 con 0x7fcf8815dc80 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf87fff640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7fcf70073a70 0x7fcf70075e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.289+0000 7fcf877fe640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf7006b8d0 0x7fcf7006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.290+0000 7fcf87fff640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7fcf70073a70 0x7fcf70075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.290+0000 7fcf877fe640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf7006b8d0 0x7fcf7006dd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf880cfb50 tx=0x7fcf78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.291+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== osd.1 v2:172.21.15.12:6808/1692589129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcf70076540 con 0x7fcf70073a70 2024-03-08T08:29:15.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.302+0000 7f4933f43640 1 -- 172.21.15.12:0/3373575779 --> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f492c1438b0 con 0x7f49100737c0 2024-03-08T08:29:15.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.303+0000 7f49227fc640 1 -- 172.21.15.12:0/3373575779 <== osd.3 v2:172.21.15.12:6810/2394545747 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f492c1438b0 con 0x7f49100737c0 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f49100737c0 msgr2=0x7f4910075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f49100737c0 0x7f4910075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f491006b5a0 msgr2=0x7f491006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f491006b5a0 0x7f491006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f491c0090b0 tx=0x7f491c03d040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f492c15dc30 msgr2=0x7f492c14f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f490ffff640 1 --2- 172.21.15.12:0/3373575779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f492c15dc30 0x7f492c14f360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f492800d530 tx=0x7f492800d9f0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.304+0000 7f49324e0640 1 -- 172.21.15.12:0/3373575779 reap_dead start 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.305+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 shutdown_connections 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.305+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 >> 172.21.15.12:0/3373575779 conn(0x7f492c06abd0 msgr2=0x7f492c137a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.306+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 shutdown_connections 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.306+0000 7f490ffff640 1 -- 172.21.15.12:0/3373575779 wait complete. 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.315+0000 7fa507284640 1 -- 172.21.15.12:0/2629982285 --> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa500071760 con 0x7fa4e4073700 2024-03-08T08:29:15.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.316+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 --> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80840e65e0 con 0x7f8070073670 2024-03-08T08:29:15.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.316+0000 7fa4f5ffb640 1 -- 172.21.15.12:0/2629982285 <== osd.0 v2:172.21.15.12:6809/1155150761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa500071760 con 0x7fa4e4073700 2024-03-08T08:29:15.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.316+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7fa4e4073700 msgr2=0x7fa4e4075b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa4e37fe640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7fa4e4073700 0x7fa4e4075b20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.316+0000 7f8081ffb640 1 -- 172.21.15.12:0/4193951548 <== osd.7 v2:172.21.15.156:6808/1139145947 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80840e65e0 con 0x7f8070073670 2024-03-08T08:29:15.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa4e406b500 msgr2=0x7fa4e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa4e37fe640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa4e406b500 0x7fa4e406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa50015e250 tx=0x7fa4f0008040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 msgr2=0x7fa50015da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa4e37fe640 1 --2- 172.21.15.12:0/2629982285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ec0039e0 0x7fa50015da30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0017e0 tx=0x7fa4fc001970 comp rx=0 tx=0).stop 2024-03-08T08:29:15.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 --> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa7c0070290 con 0x7fa7ac073720 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7f8070073670 msgr2=0x7f8070075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7f8070073670 0x7f8070075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f807006b4d0 msgr2=0x7f807006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f807006b4d0 0x7f807006d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f807c009a00 tx=0x7f807c008040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 msgr2=0x7f808404dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80920b7640 1 --2- 172.21.15.12:0/4193951548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8084037bb0 0x7f808404dad0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f808802bf30 tx=0x7f8088004730 comp rx=0 tx=0).stop 2024-03-08T08:29:15.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.317+0000 7f80918b6640 1 -- 172.21.15.12:0/4193951548 reap_dead start 2024-03-08T08:29:15.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa7bcff9640 1 -- 172.21.15.12:0/2361896558 <== osd.6 v2:172.21.15.156:6800/2822268422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa7c0070290 con 0x7fa7ac073720 2024-03-08T08:29:15.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 shutdown_connections 2024-03-08T08:29:15.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 >> 172.21.15.12:0/4193951548 conn(0x7f8084009fb0 msgr2=0x7f8084052e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 shutdown_connections 2024-03-08T08:29:15.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7f80920b7640 1 -- 172.21.15.12:0/4193951548 wait complete. 2024-03-08T08:29:15.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa505821640 1 -- 172.21.15.12:0/2629982285 reap_dead start 2024-03-08T08:29:15.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 shutdown_connections 2024-03-08T08:29:15.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 >> 172.21.15.12:0/2629982285 conn(0x7fa50006f480 msgr2=0x7fa5000b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 shutdown_connections 2024-03-08T08:29:15.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7fa7ac073720 msgr2=0x7fa7ac075b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa7a67fc640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7fa7ac073720 0x7fa7ac075b40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.318+0000 7fa4e37fe640 1 -- 172.21.15.12:0/2629982285 wait complete. 2024-03-08T08:29:15.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 --> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7228002d10 con 0x7f7224073740 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa7ac06b500 msgr2=0x7fa7ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa7ac06b500 0x7fa7ac06d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8001660 tx=0x7fa7b8009040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c00ce700 msgr2=0x7fa7c0161af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 --2- 172.21.15.12:0/2361896558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c00ce700 0x7fa7c0161af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0002230 tx=0x7fa7b000d520 comp rx=0 tx=0).stop 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7c4929640 1 -- 172.21.15.12:0/2361896558 reap_dead start 2024-03-08T08:29:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7f8eb1356640 1 -- 172.21.15.12:0/616407753 --> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8eac0749c0 con 0x7f8e980737c0 2024-03-08T08:29:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7f72367fc640 1 -- 172.21.15.12:0/3742301801 <== osd.2 v2:172.21.15.12:6800/2701699187 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7228002d10 con 0x7f7224073740 2024-03-08T08:29:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 shutdown_connections 2024-03-08T08:29:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.319+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 >> 172.21.15.12:0/2361896558 conn(0x7fa7c006db00 msgr2=0x7fa7c0165a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.374 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f8e937fe640 1 -- 172.21.15.12:0/616407753 <== osd.4 v2:172.21.15.156:6816/602210282 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8eac0749c0 con 0x7f8e980737c0 2024-03-08T08:29:15.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-08T08:29:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 shutdown_connections 2024-03-08T08:29:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7fa7a67fc640 1 -- 172.21.15.12:0/2361896558 wait complete. 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f7224073740 msgr2=0x7f7224075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f7224073740 0x7f7224075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f722406b5a0 msgr2=0x7f722406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f722406b5a0 0x7f722406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f723c009aa0 tx=0x7f723c009190 comp rx=0 tx=0).stop 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f724015fca0 msgr2=0x7f7240161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 --2- 172.21.15.12:0/3742301801 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f724015fca0 0x7f7240161400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f723000c390 tx=0x7f723000c790 comp rx=0 tx=0).stop 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7246564640 1 -- 172.21.15.12:0/3742301801 reap_dead start 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 shutdown_connections 2024-03-08T08:29:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.320+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 >> 172.21.15.12:0/3742301801 conn(0x7f72400cf9f0 msgr2=0x7f7240164d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 shutdown_connections 2024-03-08T08:29:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f7247fc7640 1 -- 172.21.15.12:0/3742301801 wait complete. 2024-03-08T08:29:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7f8e980737c0 msgr2=0x7f8e98075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7f8e980737c0 0x7f8e98075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 --> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f0802d960 con 0x7f9efc0737c0 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8e9806b5a0 msgr2=0x7f8e9806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8e9806b5a0 0x7f8e9806da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06ebf0 tx=0x7f8ea0008040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8eac06d540 msgr2=0x7f8eac159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 --2- 172.21.15.12:0/616407753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8eac06d540 0x7f8eac159ab0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c033000 tx=0x7f8e9c0046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8eab7fe640 1 -- 172.21.15.12:0/616407753 reap_dead start 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 shutdown_connections 2024-03-08T08:29:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 >> 172.21.15.12:0/616407753 conn(0x7f8eac06abd0 msgr2=0x7f8eac0cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f9ef7fff640 1 -- 172.21.15.12:0/2195287374 <== osd.5 v2:172.21.15.156:6824/1688693160 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9f0802d960 con 0x7f9efc0737c0 2024-03-08T08:29:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.321+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 shutdown_connections 2024-03-08T08:29:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f9efc0737c0 msgr2=0x7f9efc075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f9efc0737c0 0x7f9efc075be0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9efc06b5a0 msgr2=0x7f9efc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9efc06b5a0 0x7f9efc06da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f1004fd70 tx=0x7f9f1006e040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 msgr2=0x7f9f0802d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 --2- 172.21.15.12:0/2195287374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f080e00d0 0x7f9f0802d1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000dbc0 tx=0x7f9f000046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f0f7fe640 1 -- 172.21.15.12:0/2195287374 reap_dead start 2024-03-08T08:29:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f8e917fa640 1 -- 172.21.15.12:0/616407753 wait complete. 2024-03-08T08:29:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 shutdown_connections 2024-03-08T08:29:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.322+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 >> 172.21.15.12:0/2195287374 conn(0x7f9f08055c00 msgr2=0x7f9f0804ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.323+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 shutdown_connections 2024-03-08T08:29:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.323+0000 7f9f159c7640 1 -- 172.21.15.12:0/2195287374 wait complete. 2024-03-08T08:29:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.324+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 --> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf64002dd0 con 0x7fcf70073a70 2024-03-08T08:29:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.325+0000 7fcf7ffff640 1 -- 172.21.15.12:0/442498718 <== osd.1 v2:172.21.15.12:6808/1692589129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf64002dd0 con 0x7fcf70073a70 2024-03-08T08:29:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7fcf70073a70 msgr2=0x7fcf70075e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7fcf70073a70 0x7fcf70075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf7006b8d0 msgr2=0x7fcf7006dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf7006b8d0 0x7fcf7006dd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf880cfb50 tx=0x7fcf78009040 comp rx=0 tx=0).stop 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 msgr2=0x7fcf880b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 --2- 172.21.15.12:0/442498718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8815dc80 0x7fcf880b7bd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcf7400d830 tx=0x7fcf7400dcf0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf87fff640 1 -- 172.21.15.12:0/442498718 reap_dead start 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 shutdown_connections 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.326+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 >> 172.21.15.12:0/442498718 conn(0x7fcf880d07c0 msgr2=0x7fcf8815c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.327+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 shutdown_connections 2024-03-08T08:29:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.327+0000 7fcf8c9c0640 1 -- 172.21.15.12:0/442498718 wait complete. 2024-03-08T08:29:15.384 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:15.384 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-08T08:29:15.385 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:15.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-08T08:29:15.386 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:15.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-08T08:29:15.387 INFO:teuthology.orchestra.run.smithi012.stdout:60129542149 2024-03-08T08:29:15.387 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-08T08:29:15.388 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:15.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-08T08:29:15.389 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:15.389 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-08T08:29:15.390 INFO:teuthology.orchestra.run.smithi012.stdout:64424509448 2024-03-08T08:29:15.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.593+0000 7f89f469a640 1 Processor -- start 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.593+0000 7f89f469a640 1 -- start start 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.594+0000 7f89f469a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.594+0000 7f89f469a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f89ec0b3e50 0x7f89ec06abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.594+0000 7f89f469a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ec06f0e0 con 0x7f89ec162b50 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.594+0000 7f89f469a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec0b42a0 con 0x7f89ec0b3e50 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.594+0000 7f89f469a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06fc80 con 0x7f89ec162f30 2024-03-08T08:29:15.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f8a6cbbc640 1 Processor -- start 2024-03-08T08:29:15.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2436640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f89ec162b50 0x7f89ec066ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52816/0 (socket says 172.21.15.12:52816) 2024-03-08T08:29:15.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2436640 1 -- 172.21.15.12:0/1376919227 learned_addr learned my addr 172.21.15.12:0/1376919227 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2c37640 1 --2- 172.21.15.12:0/1376919227 >> v2:172.21.15.156:3300/0 conn(0x7f89ec0b3e50 0x7f89ec06abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f1c35640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f8a6cbbc640 1 -- start start 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f0a0a399640 1 Processor -- start 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f0a0a399640 1 -- start start 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2c37640 1 -- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 msgr2=0x7f89ec0b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2c37640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec0b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2c37640 1 -- 172.21.15.12:0/1376919227 >> v1:172.21.15.12:6789/0 conn(0x7f89ec162b50 legacy=0x7f89ec066ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.595+0000 7f89f2c37640 1 -- 172.21.15.12:0/1376919227 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec06f350 con 0x7f89ec0b3e50 2024-03-08T08:29:15.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.596+0000 7f301eeec640 1 Processor -- start 2024-03-08T08:29:15.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.596+0000 7f301eeec640 1 -- start start 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.596+0000 7f8a6cbbc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a60049f70 0x7f8a60051810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.596+0000 7f8a6cbbc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a6004a350 0x7f8a60052460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.596+0000 7f8a6659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a60049f70 0x7f8a60051810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a60049f70 0x7f8a60051810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57898/0 (socket says 172.21.15.12:57898) 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a65d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8a60057440 0x7f8a60051d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47114/0 (socket says 172.21.15.12:47114) 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6659c640 1 -- 172.21.15.12:0/1928264495 learned_addr learned my addr 172.21.15.12:0/1928264495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a0a399640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a0a399640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a0406e0e0 0x7f0a040b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a0a399640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040b4a10 con 0x7f0a041599f0 2024-03-08T08:29:15.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a0a399640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a040b31a0 con 0x7f0a0406e0e0 2024-03-08T08:29:15.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a0a399640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040b4cb0 con 0x7f0a0406f5d0 2024-03-08T08:29:15.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f89f2c37640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec06abd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009d90 tx=0x7f89e402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4198fdaac008dcab server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a60054580 con 0x7f8a60057440 2024-03-08T08:29:15.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f9c03004640 1 Processor -- start 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60051380 con 0x7f8a60049f70 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60007c20 con 0x7f8a6004a350 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a08936640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a041599f0 0x7f0a040714e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52826/0 (socket says 172.21.15.12:52826) 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a08936640 1 -- 172.21.15.12:0/3460359823 learned_addr learned my addr 172.21.15.12:0/3460359823 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40448/0 (socket says 172.21.15.12:40448) 2024-03-08T08:29:15.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f89f1434640 1 -- 172.21.15.12:0/1376919227 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4039070 con 0x7f89ec0b3e50 2024-03-08T08:29:15.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f89f1434640 1 -- 172.21.15.12:0/1376919227 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e402cbf0 con 0x7f89ec0b3e50 2024-03-08T08:29:15.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f89f1434640 1 -- 172.21.15.12:0/1376919227 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4034440 con 0x7f89ec0b3e50 2024-03-08T08:29:15.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f9c03004640 1 -- start start 2024-03-08T08:29:15.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7fec14d93640 1 Processor -- start 2024-03-08T08:29:15.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a66d9d640 1 --2- 172.21.15.12:0/1928264495 >> v2:172.21.15.156:3300/0 conn(0x7f8a6004a350 0x7f8a60052460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6659c640 1 -- 172.21.15.12:0/1928264495 >> v2:172.21.15.156:3300/0 conn(0x7f8a6004a350 msgr2=0x7f8a60052460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7f8a6659c640 1 --2- 172.21.15.12:0/1928264495 >> v2:172.21.15.156:3300/0 conn(0x7f8a6004a350 0x7f8a60052460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3018144110 0x7f301806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f30180cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f8a6659c640 1 -- 172.21.15.12:0/1928264495 >> v1:172.21.15.156:6789/0 conn(0x7f8a60057440 legacy=0x7f8a60051d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f3017fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f3017fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57910/0 (socket says 172.21.15.12:57910) 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f3017fff640 1 -- 172.21.15.12:0/1404204679 learned_addr learned my addr 172.21.15.12:0/1404204679 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301d489640 1 --2- 172.21.15.12:0/1404204679 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f30180cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180714e0 con 0x7f30181444f0 2024-03-08T08:29:15.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f89f469a640 1 -- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 msgr2=0x7f89ec06abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180724f0 con 0x7f3018144110 2024-03-08T08:29:15.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7fe3d8f81640 1 Processor -- start 2024-03-08T08:29:15.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3018071730 con 0x7f30180ceac0 2024-03-08T08:29:15.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f301cc88640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.156:3300/0 conn(0x7f3018144110 0x7f301806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f89f469a640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec06abd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009d90 tx=0x7f89e402be40 comp rx=0 tx=0).stop 2024-03-08T08:29:15.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f1ffac38640 1 Processor -- start 2024-03-08T08:29:15.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f8a6659c640 1 -- 172.21.15.12:0/1928264495 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60007f00 con 0x7f8a60049f70 2024-03-08T08:29:15.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.597+0000 7fec14d93640 1 -- start start 2024-03-08T08:29:15.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f9c03004640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9bfc1599f0 0x7f9bfc0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f0a03fff640 1 -- 172.21.15.12:0/3460359823 >> v2:172.21.15.12:3300/0 conn(0x7f0a0406e0e0 msgr2=0x7f0a040b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f1ffac38640 1 -- start start 2024-03-08T08:29:15.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f9c03004640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0b4cb0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f0a03fff640 1 --2- 172.21.15.12:0/3460359823 >> v2:172.21.15.12:3300/0 conn(0x7f0a0406e0e0 0x7f0a040b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f0a02ffd640 1 -- 172.21.15.12:0/3460359823 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276122305 0 0) 0x7f0a040b4a10 con 0x7f0a041599f0 2024-03-08T08:29:15.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f0a03fff640 1 -- 172.21.15.12:0/3460359823 >> v1:172.21.15.12:6789/0 conn(0x7f0a041599f0 legacy=0x7f0a040714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f0a03fff640 1 -- 172.21.15.12:0/3460359823 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040b2b50 con 0x7f0a0406f5d0 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f9c03004640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0b2b50 con 0x7f9bfc06e0e0 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7f9c03004640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bfc0b31a0 con 0x7f9bfc1599f0 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfbfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9bfc159dd0 0x7f9bfc0b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52838/0 (socket says 172.21.15.12:52838) 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfbfff640 1 -- 172.21.15.12:0/705783617 learned_addr learned my addr 172.21.15.12:0/705783617 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9c00da0640 1 --2- 172.21.15.12:0/705783617 >> v2:172.21.15.156:3300/0 conn(0x7f9bfc1599f0 0x7f9bfc0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec14d93640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec10161e80 0x7fec10162e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec14d93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec100ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec14d93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec10163370 con 0x7fec1006fa70 2024-03-08T08:29:15.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928280607 0 0) 0x7f9bfc0b2b50 con 0x7f9bfc06e0e0 2024-03-08T08:29:15.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bec003520 con 0x7f9bfc06e0e0 2024-03-08T08:29:15.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec14d93640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec10072ae0 con 0x7fec10161e80 2024-03-08T08:29:15.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec0e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec10161e80 0x7fec10162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655441522 0 0) 0x7f9bfc0b4cb0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc0b2b50 con 0x7f9bfc159dd0 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec0e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec10161e80 0x7fec10162e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57916/0 (socket says 172.21.15.12:57916) 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec14d93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec100722d0 con 0x7fec100b4610 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec0e59c640 1 -- 172.21.15.12:0/3049740706 learned_addr learned my addr 172.21.15.12:0/3049740706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec0ed9d640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec100ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9c00da0640 1 -- 172.21.15.12:0/705783617 >> v1:172.21.15.156:6789/0 conn(0x7f9bfc06e0e0 legacy=0x7f9bfc0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9c00da0640 1 -- 172.21.15.12:0/705783617 >> v1:172.21.15.12:6789/0 conn(0x7f9bfc159dd0 legacy=0x7f9bfc0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7fec0dd9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fec1006fa70 0x7fec100b3f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47120/0 (socket says 172.21.15.12:47120) 2024-03-08T08:29:15.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f9c00da0640 1 -- 172.21.15.12:0/705783617 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc0b2e30 con 0x7f9bfc1599f0 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f301d489640 1 -- 172.21.15.12:0/1404204679 >> v2:172.21.15.156:3300/0 conn(0x7f3018144110 msgr2=0x7f301806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f301d489640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.156:3300/0 conn(0x7f3018144110 0x7f301806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f301d489640 1 -- 172.21.15.12:0/1404204679 >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 msgr2=0x7f301806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f301d489640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.599+0000 7f301d489640 1 -- 172.21.15.12:0/1404204679 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3018071a10 con 0x7f30180ceac0 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f3017fff640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f301d489640 1 --2- 172.21.15.12:0/1404204679 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f30180cef30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f301806f9d0 tx=0x7f300c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c03e643fff59102f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffac38640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 0x7f1fec04fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffac38640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fec006a40 0x7f1fec006e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffac38640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec054c00 con 0x7f1fec04b3b0 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffac38640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec0062d0 con 0x7f1fec006a40 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffac38640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec050110 con 0x7f1fec034010 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ff9c36640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1fec04b3b0 0x7f1fec04f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52862/0 (socket says 172.21.15.12:52862) 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ff9c36640 1 -- 172.21.15.12:0/373835384 learned_addr learned my addr 172.21.15.12:0/373835384 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ffa437640 1 --2- 172.21.15.12:0/373835384 >> v2:172.21.15.12:3300/0 conn(0x7f1fec006a40 0x7f1fec006e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f1ff9435640 1 --2- 172.21.15.12:0/373835384 >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 0x7f1fec04fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f0a03fff640 1 --2- 172.21.15.12:0/3460359823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09f80097a0 tx=0x7f09f802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a91ce4bbab68cb5b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 -- 172.21.15.12:0/1376919227 shutdown_connections 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec06abd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 --2- 172.21.15.12:0/1376919227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec0b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 -- 172.21.15.12:0/1376919227 >> 172.21.15.12:0/1376919227 conn(0x7f89ec06db00 msgr2=0x7f89ec06dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 -- 172.21.15.12:0/1376919227 shutdown_connections 2024-03-08T08:29:15.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f89f469a640 1 -- 172.21.15.12:0/1376919227 wait complete. 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fec0e59c640 1 -- 172.21.15.12:0/3049740706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 msgr2=0x7fec100ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fec0e59c640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec100ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fec0e59c640 1 -- 172.21.15.12:0/3049740706 >> v1:172.21.15.156:6789/0 conn(0x7fec1006fa70 legacy=0x7fec100b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fec0e59c640 1 -- 172.21.15.12:0/3049740706 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec100cde20 con 0x7fec10161e80 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.598+0000 7fe3d8f81640 1 -- start start 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d8f81640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3d40b2ca0 0x7fe3d415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d8f81640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40b3210 con 0x7fe3d4157a90 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d8f81640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40b34f0 con 0x7fe3d406c2f0 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d8f81640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d40b37d0 con 0x7fe3d40b2ca0 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d1d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe3d406c2f0 0x7fe3d406ea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:47124/0 (socket says 172.21.15.12:47124) 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d1d9b640 1 -- 172.21.15.12:0/1838886496 learned_addr learned my addr 172.21.15.12:0/1838886496 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f30177fe640 1 -- 172.21.15.12:0/1404204679 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0317c0 con 0x7f30180ceac0 2024-03-08T08:29:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f0a02ffd640 1 -- 172.21.15.12:0/3460359823 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8035020 con 0x7f0a0406f5d0 2024-03-08T08:29:15.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7fe3d2d9d640 1 --2- 172.21.15.12:0/1838886496 >> v2:172.21.15.12:3300/0 conn(0x7fe3d40b2ca0 0x7fe3d415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f30177fe640 1 -- 172.21.15.12:0/1404204679 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c037070 con 0x7f30180ceac0 2024-03-08T08:29:15.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f30177fe640 1 -- 172.21.15.12:0/1404204679 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c031b80 con 0x7f30180ceac0 2024-03-08T08:29:15.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f8a6659c640 1 --2- 172.21.15.12:0/1928264495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a60051810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a600379e0 tx=0x7f8a5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce159ef7a29bfdec server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f0a02ffd640 1 -- 172.21.15.12:0/3460359823 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f8031830 con 0x7f0a0406f5d0 2024-03-08T08:29:15.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.600+0000 7f0a02ffd640 1 -- 172.21.15.12:0/3460359823 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8031c00 con 0x7f0a0406f5d0 2024-03-08T08:29:15.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f9c00da0640 1 --2- 172.21.15.12:0/705783617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc1599f0 0x7f9bfc0b3630 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0b3320 tx=0x7f9bf002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d32744de219aeb40 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723701413 0 0) 0x7fe3d40b34f0 con 0x7fe3d406c2f0 2024-03-08T08:29:15.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc003520 con 0x7fe3d406c2f0 2024-03-08T08:29:15.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f1ffa437640 1 -- 172.21.15.12:0/373835384 >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 msgr2=0x7f1fec04fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f1ffa437640 1 --2- 172.21.15.12:0/373835384 >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 0x7f1fec04fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19325590 0 0) 0x7fe3d40b3210 con 0x7fe3d4157a90 2024-03-08T08:29:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d40b34f0 con 0x7fe3d4157a90 2024-03-08T08:29:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f1ffa437640 1 -- 172.21.15.12:0/373835384 >> v1:172.21.15.12:6789/0 conn(0x7f1fec04b3b0 legacy=0x7f1fec04f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33159500 0 0) 0x7fe3bc003520 con 0x7fe3d406c2f0 2024-03-08T08:29:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 >> v2:172.21.15.12:3300/0 conn(0x7fe3d40b2ca0 msgr2=0x7fe3d415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 --2- 172.21.15.12:0/1838886496 >> v2:172.21.15.12:3300/0 conn(0x7fe3d40b2ca0 0x7fe3d415a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 >> v1:172.21.15.12:6789/0 conn(0x7fe3d4157a90 legacy=0x7fe3d406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 msgr2=0x7f30180cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f1ffa437640 1 -- 172.21.15.12:0/373835384 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec053a40 con 0x7f1fec006a40 2024-03-08T08:29:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0034070 con 0x7f9bfc1599f0 2024-03-08T08:29:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f0a0a399640 1 -- 172.21.15.12:0/3460359823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 msgr2=0x7f0a040b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3460359823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09f80097a0 tx=0x7f09f802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0030800 con 0x7f9bfc1599f0 2024-03-08T08:29:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7f9bfb7fe640 1 -- 172.21.15.12:0/705783617 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0034070 con 0x7f9bfc1599f0 2024-03-08T08:29:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40b3ab0 con 0x7fe3d406c2f0 2024-03-08T08:29:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fe3c8001670 con 0x7fe3d406c2f0 2024-03-08T08:29:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 --2- 172.21.15.12:0/1404204679 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f30180cef30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f301806f9d0 tx=0x7f300c02b860 comp rx=0 tx=0).stop 2024-03-08T08:29:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 >> v1:172.21.15.156:6789/0 conn(0x7fe3d406c2f0 legacy=0x7fe3d406ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3bc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.601+0000 7fe3d159a640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3bc004800 0x7fe3bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d159a640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc0051c0 0x7fe3bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b37d0 con 0x7fe3bc0051c0 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc00e250 con 0x7fe3bc004800 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc00e450 con 0x7fe3bc003b40 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3bc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f8a6559a640 1 -- 172.21.15.12:0/1928264495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c035020 con 0x7f8a60049f70 2024-03-08T08:29:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 shutdown_connections 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 --2- 172.21.15.12:0/1404204679 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f30180cef30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.12:3300/0 conn(0x7f30181444f0 0x7f301806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d1d9b640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3bc004800 0x7fe3bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 --2- 172.21.15.12:0/1404204679 >> v2:172.21.15.156:3300/0 conn(0x7f3018144110 0x7f301806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 >> 172.21.15.12:0/1404204679 conn(0x7f301806abd0 msgr2=0x7f301806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f8a6559a640 1 -- 172.21.15.12:0/1928264495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c031870 con 0x7f8a60049f70 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 -- 172.21.15.12:0/3460359823 shutdown_connections 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3460359823 >> v2:172.21.15.12:3300/0 conn(0x7f0a0406e0e0 0x7f0a040b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3460359823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a0406f5d0 0x7f0a040b3630 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 -- 172.21.15.12:0/3460359823 >> 172.21.15.12:0/3460359823 conn(0x7f0a0406abd0 msgr2=0x7f0a0406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f8a6559a640 1 -- 172.21.15.12:0/1928264495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c031c40 con 0x7f8a60049f70 2024-03-08T08:29:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 -- 172.21.15.12:0/3460359823 shutdown_connections 2024-03-08T08:29:15.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f89f469a640 1 Processor -- start 2024-03-08T08:29:15.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f0a0a399640 1 -- 172.21.15.12:0/3460359823 wait complete. 2024-03-08T08:29:15.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 -- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3bc004800 msgr2=0x7fe3bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fec0e59c640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec10162e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec04009870 tx=0x7fec0402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747a55f9455d4658 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3bc004800 0x7fe3bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 -- 172.21.15.12:0/1838886496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc0051c0 msgr2=0x7fe3bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc0051c0 0x7fe3bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d2d9d640 1 -- 172.21.15.12:0/1838886496 reap_dead start 2024-03-08T08:29:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d259c640 1 -- 172.21.15.12:0/1838886496 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40b3ab0 con 0x7fe3bc003b40 2024-03-08T08:29:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f89f469a640 1 -- start start 2024-03-08T08:29:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fe3d1d9b640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3bc004800 0x7fe3bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7f1ff9435640 1 --2- 172.21.15.12:0/373835384 >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 0x7f1fec04fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fec0d59a640 1 -- 172.21.15.12:0/3049740706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04034070 con 0x7fec10161e80 2024-03-08T08:29:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fec0d59a640 1 -- 172.21.15.12:0/3049740706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec04030840 con 0x7fec10161e80 2024-03-08T08:29:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.602+0000 7fec0d59a640 1 -- 172.21.15.12:0/3049740706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04030c10 con 0x7fec10161e80 2024-03-08T08:29:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 shutdown_connections 2024-03-08T08:29:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 msgr2=0x7f8a60051810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1928264495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a60051810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a600379e0 tx=0x7f8a5c02b910 comp rx=0 tx=0).stop 2024-03-08T08:29:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f469a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f469a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 0x7f89ec14aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7fe3d259c640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3bc003f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001fc0 tx=0x7fe3bc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8a2af94fee2ba3d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 shutdown_connections 2024-03-08T08:29:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1928264495 >> v2:172.21.15.156:3300/0 conn(0x7f8a6004a350 0x7f8a60052460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1928264495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a60051810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f9c03004640 1 -- 172.21.15.12:0/705783617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc1599f0 msgr2=0x7f9bfc0b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 >> 172.21.15.12:0/1928264495 conn(0x7f8a60020000 msgr2=0x7f8a60056d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f9c03004640 1 --2- 172.21.15.12:0/705783617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc1599f0 0x7f9bfc0b3630 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0b3320 tx=0x7f9bf002b940 comp rx=0 tx=0).stop 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f1ffa437640 1 --2- 172.21.15.12:0/373835384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec006a40 0x7f1fec006e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1fec006450 tx=0x7f1fe402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33ecd5ef342c61b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f1ff8c34640 1 -- 172.21.15.12:0/373835384 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4035020 con 0x7f1fec006a40 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f1ff8c34640 1 -- 172.21.15.12:0/373835384 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4031640 con 0x7f1fec006a40 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f2436640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f2436640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53472/0 (socket says 172.21.15.12:53472) 2024-03-08T08:29:15.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f2436640 1 -- 172.21.15.12:0/855269246 learned_addr learned my addr 172.21.15.12:0/855269246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f301eeec640 1 -- 172.21.15.12:0/1404204679 wait complete. 2024-03-08T08:29:15.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.603+0000 7f89f1c35640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 0x7f89ec14aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f9c03004640 1 -- 172.21.15.12:0/705783617 shutdown_connections 2024-03-08T08:29:15.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f469a640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec14eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 -- 172.21.15.12:0/3049740706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 msgr2=0x7fec10162e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f9c03004640 1 --2- 172.21.15.12:0/705783617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc1599f0 0x7f9bfc0b3630 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f9c03004640 1 -- 172.21.15.12:0/705783617 >> 172.21.15.12:0/705783617 conn(0x7f9bfc06abd0 msgr2=0x7f9bfc06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec10162e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec04009870 tx=0x7fec0402b980 comp rx=0 tx=0).stop 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc001be0 con 0x7fe3bc003b40 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc006d70 con 0x7fe3bc003b40 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fe3d159a640 1 -- 172.21.15.12:0/1838886496 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc003bb0 con 0x7fe3bc003b40 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec1633b0 con 0x7f89ec162b50 2024-03-08T08:29:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06f910 con 0x7f89ec0b3e50 2024-03-08T08:29:15.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f0a0a399640 1 Processor -- start 2024-03-08T08:29:15.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec070110 con 0x7f89ec162f30 2024-03-08T08:29:15.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f2c37640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec14eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 -- 172.21.15.12:0/3049740706 shutdown_connections 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec100ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 --2- 172.21.15.12:0/3049740706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec10162e30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 -- 172.21.15.12:0/3049740706 >> 172.21.15.12:0/3049740706 conn(0x7fec100717b0 msgr2=0x7fec10071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f9c03004640 1 -- 172.21.15.12:0/705783617 shutdown_connections 2024-03-08T08:29:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 shutdown_connections 2024-03-08T08:29:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f0a0a399640 1 -- start start 2024-03-08T08:29:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1928264495 wait complete. 2024-03-08T08:29:15.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f1ffac38640 1 -- 172.21.15.12:0/373835384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec006a40 msgr2=0x7f1fec006e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f1ff8c34640 1 -- 172.21.15.12:0/373835384 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4035020 con 0x7f1fec006a40 2024-03-08T08:29:15.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f1ffac38640 1 --2- 172.21.15.12:0/373835384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec006a40 0x7f1fec006e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1fec006450 tx=0x7f1fe402b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f1c35640 1 -- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 msgr2=0x7f89ec14eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7fec14d93640 1 -- 172.21.15.12:0/3049740706 shutdown_connections 2024-03-08T08:29:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f1c35640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec14eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f1ffac38640 1 -- 172.21.15.12:0/373835384 shutdown_connections 2024-03-08T08:29:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f1c35640 1 -- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 msgr2=0x7f89ec14a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f1c35640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.604+0000 7f89f1c35640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec15cba0 con 0x7f89ec162b50 2024-03-08T08:29:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f89f2436640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fec14d93640 1 -- 172.21.15.12:0/3049740706 wait complete. 2024-03-08T08:29:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 -- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 msgr2=0x7fe3bc003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f9c03004640 1 -- 172.21.15.12:0/705783617 wait complete. 2024-03-08T08:29:15.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3bc003f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001fc0 tx=0x7fe3bc005830 comp rx=0 tx=0).stop 2024-03-08T08:29:15.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 -- 172.21.15.12:0/1838886496 shutdown_connections 2024-03-08T08:29:15.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/1838886496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3bc003f40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 -- 172.21.15.12:0/1838886496 >> 172.21.15.12:0/1838886496 conn(0x7fe3d4071a30 msgr2=0x7fe3d4071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f89f1c35640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 0x7f89ec14aaa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f89e800d550 tx=0x7f89e800da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 -- 172.21.15.12:0/1838886496 shutdown_connections 2024-03-08T08:29:15.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f1ffac38640 1 --2- 172.21.15.12:0/373835384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec006a40 0x7f1fec006e70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f1ffac38640 1 --2- 172.21.15.12:0/373835384 >> v2:172.21.15.156:3300/0 conn(0x7f1fec034010 0x7f1fec04fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f1ffac38640 1 -- 172.21.15.12:0/373835384 >> 172.21.15.12:0/373835384 conn(0x7f1fec00a3f0 msgr2=0x7f1fec00a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:15.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f0a0a399640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80125f0 con 0x7f89ec162b50 2024-03-08T08:29:15.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec14b1f0 con 0x7f89ec162b50 2024-03-08T08:29:15.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7f0a0a399640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57958/0 (socket says 172.21.15.12:57958) 2024-03-08T08:29:15.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f301eeec640 1 Processor -- start 2024-03-08T08:29:15.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53480/0 (socket says 172.21.15.12:53480) 2024-03-08T08:29:15.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f301eeec640 1 -- start start 2024-03-08T08:29:15.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a0a399640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 0x7f0a0415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a03fff640 1 -- 172.21.15.12:0/3033556726 learned_addr learned my addr 172.21.15.12:0/3033556726 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a08936640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 0x7f0a0415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04071d70 con 0x7f0a0406e0e0 2024-03-08T08:29:15.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f9c03004640 1 Processor -- start 2024-03-08T08:29:15.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec14b7b0 con 0x7f89ec162b50 2024-03-08T08:29:15.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8019070 con 0x7f89ec162b50 2024-03-08T08:29:15.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f9c03004640 1 -- start start 2024-03-08T08:29:15.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f8a6cbbc640 1 Processor -- start 2024-03-08T08:29:15.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7fec14d93640 1 Processor -- start 2024-03-08T08:29:15.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.605+0000 7fe3d8f81640 1 -- 172.21.15.12:0/1838886496 wait complete. 2024-03-08T08:29:15.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.606+0000 7f8a6cbbc640 1 -- start start 2024-03-08T08:29:15.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80129b0 con 0x7f89ec162b50 2024-03-08T08:29:15.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7f301eeec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f301813db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04071bf0 con 0x7f0a0406f5d0 2024-03-08T08:29:15.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04159e50 con 0x7f0a041599f0 2024-03-08T08:29:15.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7fe3d8f81640 1 Processor -- start 2024-03-08T08:29:15.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7fe3d8f81640 1 -- start start 2024-03-08T08:29:15.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.607+0000 7fec14d93640 1 -- start start 2024-03-08T08:29:15.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bfc06e0e0 0x7f9bfc15bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bfc1599f0 0x7f9bfc1f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f301eeec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f8a6cbbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec1006fa70 0x7fec102004d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec10164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc06f5d0 con 0x7f9bfc06e0e0 2024-03-08T08:29:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc06f750 con 0x7f9bfc159dd0 2024-03-08T08:29:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c03004640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc164b80 con 0x7f9bfc1599f0 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f301cc88640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f301813db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10072520 con 0x7fec10161e80 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10162260 con 0x7fec1006fa70 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec14d93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec10072800 con 0x7fec100b4610 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec0dd9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec10164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec0ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f1ffac38640 1 -- 172.21.15.12:0/373835384 shutdown_connections 2024-03-08T08:29:15.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c015a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c015a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53486/0 (socket says 172.21.15.12:53486) 2024-03-08T08:29:15.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f9c015a1640 1 -- 172.21.15.12:0/1122732445 learned_addr learned my addr 172.21.15.12:0/1122732445 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e8012b50 con 0x7f89ec162b50 2024-03-08T08:29:15.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec0ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57994/0 (socket says 172.21.15.12:57994) 2024-03-08T08:29:15.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f3017fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f3017fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57978/0 (socket says 172.21.15.12:57978) 2024-03-08T08:29:15.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f301cc88640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f301813db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40520/0 (socket says 172.21.15.12:40520) 2024-03-08T08:29:15.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fe3d8f81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3d40af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f3017fff640 1 -- 172.21.15.12:0/1571168328 learned_addr learned my addr 172.21.15.12:0/1571168328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fe3d8f81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 0x7fe3d4154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 0x7f301815d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fe3d8f81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 0x7fe3d40ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.608+0000 7fec0ed9d640 1 -- 172.21.15.12:0/3429573914 learned_addr learned my addr 172.21.15.12:0/3429573914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d8f81640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b4f70 con 0x7fe3d40b0b20 2024-03-08T08:29:15.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58002/0 (socket says 172.21.15.12:58002) 2024-03-08T08:29:15.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30180cf470 con 0x7f3018144110 2024-03-08T08:29:15.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3d40af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 0x7fe3d4154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d259c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3d40af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40542/0 (socket says 172.21.15.12:40542) 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d259c640 1 -- 172.21.15.12:0/4248833206 learned_addr learned my addr 172.21.15.12:0/4248833206 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b4df0 con 0x7fe3d40cc760 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d415a4d0 con 0x7fe3bc003b40 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f301d489640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 0x7f301815d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 -- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 msgr2=0x7f0a0415b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 0x7f0a0415b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 -- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 msgr2=0x7f0a041e2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04152e90 con 0x7f0a0406e0e0 2024-03-08T08:29:15.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3018071fd0 con 0x7f301815a310 2024-03-08T08:29:15.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30180cf5f0 con 0x7f30180ceac0 2024-03-08T08:29:15.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6cbbc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a6004a350 0x7f8a6004de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a037fe640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a08936640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 0x7f0a0415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f89e2ffd640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 0x7f89d006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fe3d2d9d640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 0x7fe3d40ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f9c015a1640 1 -- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bfc1599f0 msgr2=0x7f9bfc1f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f9c015a1640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bfc1599f0 0x7f9bfc1f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f9c015a1640 1 -- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bfc06e0e0 msgr2=0x7f9bfc15bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f9c015a1640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bfc06e0e0 0x7f9bfc15bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f9c015a1640 1 -- 172.21.15.12:0/1122732445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc15c9a0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d4005290 con 0x7f89ec162b50 2024-03-08T08:29:15.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f1ffac38640 1 -- 172.21.15.12:0/373835384 wait complete. 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f89f2436640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 0x7f89d006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fec0ed9d640 1 -- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 msgr2=0x7fec10164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89e808a320 con 0x7f89ec162b50 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fec0ed9d640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec10164ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fec0ed9d640 1 -- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec1006fa70 msgr2=0x7fec102004d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-08T08:29:15.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fec0ed9d640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec1006fa70 0x7fec102004d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7fec0ed9d640 1 -- 172.21.15.12:0/3429573914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1014c6c0 con 0x7fec10161e80 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6cbbc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 0x7f8a600da1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6cbbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600529a0 con 0x7f8a60049f70 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6cbbc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60038540 con 0x7f8a60057440 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6cbbc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60052b20 con 0x7f8a6004a350 2024-03-08T08:29:15.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fec0ed9d640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec0800efc0 tx=0x7fec0800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 -- 172.21.15.12:0/1000007578 learned_addr learned my addr 172.21.15.12:0/1000007578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a65d9b640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a6004a350 0x7f8a6004de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a66d9d640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 0x7f8a600da1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 -- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a6004a350 msgr2=0x7f8a6004de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a6004a350 0x7f8a6004de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 -- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 msgr2=0x7f8a600da1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 -- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 msgr2=0x7fe3d40af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3d40af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 -- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 msgr2=0x7fe3d40ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 0x7fe3d40ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d4154dd0 con 0x7fe3d40b0b20 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f89f2436640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 0x7f89d006da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89dc007d50 tx=0x7f89dc007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d2d9d640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 0x7fe3d40ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f0a03fff640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098c0 tx=0x7f09f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 0x7f8a600da1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.609+0000 7f8a6659c640 1 -- 172.21.15.12:0/1000007578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6004e340 con 0x7f8a60049f70 2024-03-08T08:29:15.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f8a66d9d640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 0x7f8a600da1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:15.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f8a6659c640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7fe3d1d9b640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 0x7fe3d4154890 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40693a0 tx=0x7fe3c8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f9c015a1640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c750 tx=0x7f9bf400cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 -- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 msgr2=0x7f301813db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f301813db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 -- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 msgr2=0x7f301815d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 0x7f301815d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301815ae70 con 0x7f3018144110 2024-03-08T08:29:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f301d489640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 0x7f301815d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf400ce60 con 0x7f9bfc159dd0 2024-03-08T08:29:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f3017fff640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f300800c7f0 tx=0x7f300800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.610+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf4003b70 con 0x7f9bfc159dd0 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.611+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc15cc80 con 0x7f9bfc159dd0 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.611+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf400a850 con 0x7f9bfc159dd0 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.611+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc15c170 con 0x7f9bfc159dd0 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.611+0000 7f1ffac38640 1 Processor -- start 2024-03-08T08:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.611+0000 7f1ffac38640 1 -- start start 2024-03-08T08:29:15.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc13e040 con 0x7f9bfc159dd0 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fec006a40 0x7f1fec037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fec04b3b0 0x7f1fec0e9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec04b790 con 0x7f1fec034010 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec006520 con 0x7f1fec04b3b0 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ff9435640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffac38640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec054820 con 0x7f1fec006a40 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ff9435640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58022/0 (socket says 172.21.15.12:58022) 2024-03-08T08:29:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ff9c36640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fec006a40 0x7f1fec037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ff9435640 1 -- 172.21.15.12:0/3191132171 learned_addr learned my addr 172.21.15.12:0/3191132171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.612+0000 7f1ffa437640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fec04b3b0 0x7f1fec0e9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 -- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fec006a40 msgr2=0x7f1fec037af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fec006a40 0x7f1fec037af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 -- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fec04b3b0 msgr2=0x7f1fec0e9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fec04b3b0 0x7f1fec0e9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 -- 172.21.15.12:0/3191132171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0d8980 con 0x7f1fec034010 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bf400a9f0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f1ff9435640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40693a0 tx=0x7f1ff4074500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f9bf97fa640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bf408a1d0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.613+0000 7f9c00da0640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.614+0000 7f9c00da0640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 0x7f9be006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f803ba80 con 0x7f0a0406e0e0 2024-03-08T08:29:15.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008015070 con 0x7f3018144110 2024-03-08T08:29:15.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4094640 con 0x7f1fec034010 2024-03-08T08:29:15.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04153170 con 0x7f0a0406e0e0 2024-03-08T08:29:15.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8007d10 con 0x7fe3d40b0b20 2024-03-08T08:29:15.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301815b110 con 0x7f3018144110 2024-03-08T08:29:15.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301815b6d0 con 0x7f3018144110 2024-03-08T08:29:15.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04153730 con 0x7f0a0406e0e0 2024-03-08T08:29:15.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d40afa80 con 0x7fe3d40b0b20 2024-03-08T08:29:15.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f89e8058b30 con 0x7f89ec162b50 2024-03-08T08:29:15.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08018020 con 0x7fec10161e80 2024-03-08T08:29:15.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec08014480 con 0x7fec10161e80 2024-03-08T08:29:15.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08014870 con 0x7fec10161e80 2024-03-08T08:29:15.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f803bc20 con 0x7f0a0406e0e0 2024-03-08T08:29:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d40b0040 con 0x7fe3d40b0b20 2024-03-08T08:29:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec0380d0 con 0x7f1fec034010 2024-03-08T08:29:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff40947e0 con 0x7f1fec034010 2024-03-08T08:29:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.615+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec038690 con 0x7f1fec034010 2024-03-08T08:29:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80453f0 con 0x7f0a0406e0e0 2024-03-08T08:29:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1014c9a0 con 0x7fec10161e80 2024-03-08T08:29:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c040070 con 0x7f8a60049f70 2024-03-08T08:29:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1014cf60 con 0x7fec10161e80 2024-03-08T08:29:15.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c03a420 con 0x7f8a60049f70 2024-03-08T08:29:15.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c8018070 con 0x7fe3d40b0b20 2024-03-08T08:29:15.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c800fb50 con 0x7fe3d40b0b20 2024-03-08T08:29:15.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e4005290 con 0x7f0a0406e0e0 2024-03-08T08:29:15.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3008007820 con 0x7f3018144110 2024-03-08T08:29:15.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008007bd0 con 0x7f3018144110 2024-03-08T08:29:15.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.616+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4094b90 con 0x7f1fec034010 2024-03-08T08:29:15.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.617+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30180749c0 con 0x7f3018144110 2024-03-08T08:29:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.617+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a600da6f0 con 0x7f8a60049f70 2024-03-08T08:29:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.617+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec005380 con 0x7f1fec034010 2024-03-08T08:29:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.617+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d40749c0 con 0x7fe3d40b0b20 2024-03-08T08:29:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.618+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c035040 con 0x7f8a60049f70 2024-03-08T08:29:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.618+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a600dcf30 con 0x7f8a60049f70 2024-03-08T08:29:15.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.618+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec100731f0 con 0x7fec10161e80 2024-03-08T08:29:15.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.618+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9bf4058fd0 con 0x7f9bfc159dd0 2024-03-08T08:29:15.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.619+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec0801b020 con 0x7fec10161e80 2024-03-08T08:29:15.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.619+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3c800fcf0 con 0x7fe3d40b0b20 2024-03-08T08:29:15.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a600057a0 con 0x7f8a60049f70 2024-03-08T08:29:15.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.619+0000 7febfeffd640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 0x7febf006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7fec0e59c640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 0x7febf006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec08089550 con 0x7fec10161e80 2024-03-08T08:29:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7fe3c2ffd640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7fe3d259c640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ff409e3f0 con 0x7f1fec034010 2024-03-08T08:29:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3c808ab40 con 0x7fe3d40b0b20 2024-03-08T08:29:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7f1fea7fc640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7fe3d259c640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001e30 tx=0x7fe3cc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.620+0000 7fec0e59c640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 0x7febf006d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec04036000 tx=0x7fec0403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ff40d1750 con 0x7f1fec034010 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7f1ff9c36640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.621+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a5c04b060 con 0x7f8a60049f70 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f8a56ffd640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 0x7f8a4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09f804c060 con 0x7f0a0406e0e0 2024-03-08T08:29:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f0a00ff9640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 0x7f09f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3008018070 con 0x7f3018144110 2024-03-08T08:29:15.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f30157fa640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 0x7f300406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:15.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f300808a590 con 0x7f3018144110 2024-03-08T08:29:15.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f8a65d9b640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 0x7f8a4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f301cc88640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 0x7f300406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f0a037fe640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 0x7f09f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:15.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.622+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09f80ac000 con 0x7f0a0406e0e0 2024-03-08T08:29:15.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f8a65d9b640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 0x7f8a4806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006fd0 tx=0x7f8a58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f0a037fe640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 0x7f09f006d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f4006fd0 tx=0x7f09f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f1ff9c36640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 0x7f1fd806d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0005880 tx=0x7f1ff0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.623+0000 7f301cc88640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 0x7f300406d9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f30100074a0 tx=0x7f301000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.626+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a5c0788f0 con 0x7f8a60049f70 2024-03-08T08:29:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.626+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec08022030 con 0x7fec10161e80 2024-03-08T08:29:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.631+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3c8059aa0 con 0x7fe3d40b0b20 2024-03-08T08:29:15.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.640+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ff40d36b0 con 0x7f1fec034010 2024-03-08T08:29:15.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.641+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f09f8040070 con 0x7f0a0406e0e0 2024-03-08T08:29:15.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.648+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3008059580 con 0x7f3018144110 2024-03-08T08:29:15.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:15.658+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a5c07af50 con 0x7f8a60049f70 2024-03-08T08:29:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.001+0000 7f89f469a640 1 -- 172.21.15.12:0/855269246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f89d4005580 con 0x7f89ec162b50 2024-03-08T08:29:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.001+0000 7f89e2ffd640 1 -- 172.21.15.12:0/855269246 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f89e805cbd0 con 0x7f89ec162b50 2024-03-08T08:29:16.003 INFO:teuthology.orchestra.run.smithi012.stdout:60129542148 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.005+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 msgr2=0x7f89d006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.005+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 0x7f89d006da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89dc007d50 tx=0x7f89dc007ca0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.005+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 msgr2=0x7f89ec14aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.005+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 0x7f89ec14aaa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f89e800d550 tx=0x7f89e800da10 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.005+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 shutdown_connections 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.006+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f89d006b5a0 0x7f89d006da60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.006+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89ec162f30 0x7f89ec14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.006+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec162b50 0x7f89ec14aaa0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.006+0000 7f89e0ff9640 1 --2- 172.21.15.12:0/855269246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89ec0b3e50 0x7f89ec14a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.006+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 >> 172.21.15.12:0/855269246 conn(0x7f89ec06db00 msgr2=0x7f89ec0b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.007+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 shutdown_connections 2024-03-08T08:29:16.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.007+0000 7f89e0ff9640 1 -- 172.21.15.12:0/855269246 wait complete. 2024-03-08T08:29:16.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2024-03-08T08:29:16.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.035+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9bfc0749c0 con 0x7f9bfc159dd0 2024-03-08T08:29:16.037 INFO:teuthology.orchestra.run.smithi012.stdout:64424509446 2024-03-08T08:29:16.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.036+0000 7f9bf97fa640 1 -- 172.21.15.12:0/1122732445 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9bf40588d0 con 0x7f9bfc159dd0 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.038+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 msgr2=0x7f9be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.038+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 0x7f9be006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf003b040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.038+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 msgr2=0x7f9bfc1f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.038+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c750 tx=0x7f9bf400cc10 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 shutdown_connections 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bfc159dd0 0x7f9bfc1f5b50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9bfc1599f0 0x7f9bfc1f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 --2- 172.21.15.12:0/1122732445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bfc06e0e0 0x7f9bfc15bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 >> 172.21.15.12:0/1122732445 conn(0x7f9bfc06abd0 msgr2=0x7f9bfc0b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 shutdown_connections 2024-03-08T08:29:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.039+0000 7f9c03004640 1 -- 172.21.15.12:0/1122732445 wait complete. 2024-03-08T08:29:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.041+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f30180cbee0 con 0x7f3018144110 2024-03-08T08:29:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.041+0000 7f30157fa640 1 -- 172.21.15.12:0/1571168328 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3008058e80 con 0x7f3018144110 2024-03-08T08:29:16.043 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.043+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe3d4156300 con 0x7fe3d40b0b20 2024-03-08T08:29:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.043+0000 7fe3c2ffd640 1 -- 172.21.15.12:0/4248833206 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe3c80593a0 con 0x7fe3d40b0b20 2024-03-08T08:29:16.044 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:16.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.044+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 msgr2=0x7f300406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.044+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 0x7f300406d9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f30100074a0 tx=0x7f301000b040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.044+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 msgr2=0x7f3018142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.044+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f300800c7f0 tx=0x7f300800ccb0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 shutdown_connections 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f300406b500 0x7f300406d9c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f301815a310 0x7f301815d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3018144110 0x7f3018142af0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 --2- 172.21.15.12:0/1571168328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30180ceac0 0x7f301813db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 >> 172.21.15.12:0/1571168328 conn(0x7f301806abd0 msgr2=0x7f30180ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 shutdown_connections 2024-03-08T08:29:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.045+0000 7f301eeec640 1 -- 172.21.15.12:0/1571168328 wait complete. 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 msgr2=0x7fe3b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001e30 tx=0x7fe3cc030000 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 msgr2=0x7fe3d4154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 0x7fe3d4154890 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40693a0 tx=0x7fe3c8007720 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 shutdown_connections 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3d40cc760 0x7fe3d40ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d40b0b20 0x7fe3d4154890 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 --2- 172.21.15.12:0/4248833206 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3bc003b40 0x7fe3d40af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.046+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 >> 172.21.15.12:0/4248833206 conn(0x7fe3d4071a30 msgr2=0x7fe3d406fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.047+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 shutdown_connections 2024-03-08T08:29:16.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.047+0000 7fe3d8f81640 1 -- 172.21.15.12:0/4248833206 wait complete. 2024-03-08T08:29:16.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.050+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fec100af740 con 0x7fec10161e80 2024-03-08T08:29:16.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.051+0000 7febfeffd640 1 -- 172.21.15.12:0/3429573914 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec08010040 con 0x7fec10161e80 2024-03-08T08:29:16.052 INFO:teuthology.orchestra.run.smithi012.stdout:60129542149 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.054+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 msgr2=0x7febf006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.054+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 0x7febf006d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec04036000 tx=0x7fec0403b040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.054+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 msgr2=0x7fec101653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.054+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec0800efc0 tx=0x7fec0800c490 comp rx=0 tx=0).stop 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 shutdown_connections 2024-03-08T08:29:16.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7febf006b520 0x7febf006d9e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec10161e80 0x7fec101653e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec100b4610 0x7fec10164ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 --2- 172.21.15.12:0/3429573914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec1006fa70 0x7fec102004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 >> 172.21.15.12:0/3429573914 conn(0x7fec100717b0 msgr2=0x7fec10162b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 shutdown_connections 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.055+0000 7fec14d93640 1 -- 172.21.15.12:0/3429573914 wait complete. 2024-03-08T08:29:16.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.056+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f09e4005850 con 0x7f0a0406e0e0 2024-03-08T08:29:16.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.057+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1fec0ea530 con 0x7f1fec034010 2024-03-08T08:29:16.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509446 for osd.1 2024-03-08T08:29:16.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.058+0000 7f0a00ff9640 1 -- 172.21.15.12:0/3033556726 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09f804b2c0 con 0x7f0a0406e0e0 2024-03-08T08:29:16.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.058+0000 7f1fea7fc640 1 -- 172.21.15.12:0/3191132171 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ff40d7750 con 0x7f1fec034010 2024-03-08T08:29:16.060 INFO:teuthology.orchestra.run.smithi012.stdout:60129542149 2024-03-08T08:29:16.060 INFO:teuthology.orchestra.run.smithi012.stdout:60129542149 2024-03-08T08:29:16.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 msgr2=0x7f09f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 0x7f09f006d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f4006fd0 tx=0x7f09f4008040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 msgr2=0x7f0a04152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098c0 tx=0x7f09f80046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 shutdown_connections 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f09f006b4d0 0x7f09f006d990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a041599f0 0x7f0a0415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a0406f5d0 0x7f0a041e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 --2- 172.21.15.12:0/3033556726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a0406e0e0 0x7f0a04152790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.060+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 >> 172.21.15.12:0/3033556726 conn(0x7f0a0406abd0 msgr2=0x7f0a040aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 shutdown_connections 2024-03-08T08:29:16.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f0a0a399640 1 -- 172.21.15.12:0/3033556726 wait complete. 2024-03-08T08:29:16.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 msgr2=0x7f1fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 0x7f1fd806d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0005880 tx=0x7f1ff0009040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 msgr2=0x7f1fec0d8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40693a0 tx=0x7f1ff4074500 comp rx=0 tx=0).stop 2024-03-08T08:29:16.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 shutdown_connections 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fec04b3b0 0x7f1fec0e9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec034010 0x7f1fec0d8440 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 --2- 172.21.15.12:0/3191132171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fec006a40 0x7f1fec037af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 >> 172.21.15.12:0/3191132171 conn(0x7f1fec00a3f0 msgr2=0x7f1fec034990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 shutdown_connections 2024-03-08T08:29:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.061+0000 7f1ffac38640 1 -- 172.21.15.12:0/3191132171 wait complete. 2024-03-08T08:29:16.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.3 2024-03-08T08:29:16.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.5 2024-03-08T08:29:16.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.066+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8a60052ef0 con 0x7f8a60049f70 2024-03-08T08:29:16.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.067+0000 7f8a56ffd640 1 -- 172.21.15.12:0/1000007578 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a5c07a850 con 0x7f8a60049f70 2024-03-08T08:29:16.069 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:16.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 msgr2=0x7f8a4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 0x7f8a4806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006fd0 tx=0x7f8a58008040 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 msgr2=0x7f8a600d99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c0046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 shutdown_connections 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8a4806b500 0x7f8a4806d9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a60057440 0x7f8a600da1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a6004a350 0x7f8a6004de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 --2- 172.21.15.12:0/1000007578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a60049f70 0x7f8a600d99d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 >> 172.21.15.12:0/1000007578 conn(0x7f8a60020000 msgr2=0x7f8a6000d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 shutdown_connections 2024-03-08T08:29:16.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:16.069+0000 7f8a6cbbc640 1 -- 172.21.15.12:0/1000007578 wait complete. 2024-03-08T08:29:16.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2024-03-08T08:29:16.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.7 2024-03-08T08:29:16.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-03-08T08:29:16.082 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.0 2024-03-08T08:29:17.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-08T08:29:17.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-08T08:29:17.067 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-08T08:29:17.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-08T08:29:17.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-08T08:29:17.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-08T08:29:17.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-08T08:29:17.083 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-08T08:29:17.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.164+0000 7f0567fff640 1 Processor -- start 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.165+0000 7f0567fff640 1 -- start start 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.166+0000 7f0567fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0560048c30 0x7f0560049060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.166+0000 7f0567fff640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0560009560 con 0x7f0560009180 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.166+0000 7f0567fff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0560035960 con 0x7f056003aa10 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.166+0000 7f0567fff640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05600495a0 con 0x7f0560048c30 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.166+0000 7f05667fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0560048c30 0x7f0560049060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.167+0000 7f05667fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0560048c30 0x7f0560049060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58032/0 (socket says 172.21.15.12:58032) 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.167+0000 7f05667fc640 1 -- 172.21.15.12:0/602711185 learned_addr learned my addr 172.21.15.12:0/602711185 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.167+0000 7f05667fc640 1 -- 172.21.15.12:0/602711185 >> v1:172.21.15.156:6789/0 conn(0x7f056003aa10 legacy=0x7f056003ae20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T08:29:17.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.167+0000 7f05667fc640 1 -- 172.21.15.12:0/602711185 >> v1:172.21.15.12:6789/0 conn(0x7f0560009180 legacy=0x7f0560052ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.167+0000 7f05667fc640 1 -- 172.21.15.12:0/602711185 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560035c20 con 0x7f0560048c30 2024-03-08T08:29:17.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.168+0000 7f05667fc640 1 --2- 172.21.15.12:0/602711185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560049060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f056802d3c0 tx=0x7f056806aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b4624a0c71f3db server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.169+0000 7f0565ffb640 1 -- 172.21.15.12:0/602711185 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05680ac750 con 0x7f0560048c30 2024-03-08T08:29:17.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.169+0000 7f0565ffb640 1 -- 172.21.15.12:0/602711185 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05680ac8f0 con 0x7f0560048c30 2024-03-08T08:29:17.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.169+0000 7f0565ffb640 1 -- 172.21.15.12:0/602711185 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05680acd60 con 0x7f0560048c30 2024-03-08T08:29:17.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.170+0000 7f0567fff640 1 -- 172.21.15.12:0/602711185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 msgr2=0x7f0560049060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.170+0000 7f0567fff640 1 --2- 172.21.15.12:0/602711185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560049060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f056802d3c0 tx=0x7f056806aa90 comp rx=0 tx=0).stop 2024-03-08T08:29:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.170+0000 7f0567fff640 1 -- 172.21.15.12:0/602711185 shutdown_connections 2024-03-08T08:29:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.170+0000 7f0567fff640 1 --2- 172.21.15.12:0/602711185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560049060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.170+0000 7f0567fff640 1 -- 172.21.15.12:0/602711185 >> 172.21.15.12:0/602711185 conn(0x7f056001ef30 msgr2=0x7f0560055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.171+0000 7f0567fff640 1 -- 172.21.15.12:0/602711185 shutdown_connections 2024-03-08T08:29:17.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.171+0000 7f0567fff640 1 -- 172.21.15.12:0/602711185 wait complete. 2024-03-08T08:29:17.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.173+0000 7f0567fff640 1 Processor -- start 2024-03-08T08:29:17.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.173+0000 7f0567fff640 1 -- start start 2024-03-08T08:29:17.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.175+0000 7f0567fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 0x7f0560185640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.175+0000 7f0567fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.175+0000 7f0567fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560033a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.175+0000 7f0566ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 0x7f0560185640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.175+0000 7f05667fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f05667fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40566/0 (socket says 172.21.15.12:40566) 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f0566ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 0x7f0560185640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53526/0 (socket says 172.21.15.12:53526) 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f05667fc640 1 -- 172.21.15.12:0/664874018 learned_addr learned my addr 172.21.15.12:0/664874018 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05600364c0 con 0x7f0560048c30 2024-03-08T08:29:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05600361e0 con 0x7f0560009180 2024-03-08T08:29:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05600367a0 con 0x7f056003aa10 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 -- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 msgr2=0x7f0560185640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 0x7f0560185640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 -- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 msgr2=0x7f0560033a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.176+0000 7f05677fe640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560033a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560033a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05600e9170 con 0x7f056003aa10 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05677fe640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560033a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.177+0000 7f05667fc640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f05680508c0 tx=0x7f0568062a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.178+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05680bb070 con 0x7f056003aa10 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.178+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05600e9410 con 0x7f056003aa10 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.178+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05600e99d0 con 0x7f056003aa10 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.178+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0568067250 con 0x7f056003aa10 2024-03-08T08:29:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.178+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568067600 con 0x7f056003aa10 2024-03-08T08:29:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.181+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05680be070 con 0x7f056003aa10 2024-03-08T08:29:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.181+0000 7f054ffff640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 0x7f055006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.182+0000 7f0566ffd640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 0x7f055006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.183+0000 7f0566ffd640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 0x7f055006dba0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0558004fb0 tx=0x7f0558007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.183+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0568126b70 con 0x7f056003aa10 2024-03-08T08:29:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.185+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0548005290 con 0x7f056003aa10 2024-03-08T08:29:17.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.191+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f05680c5070 con 0x7f056003aa10 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.218+0000 7fadc1dc7640 1 Processor -- start 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.218+0000 7fadc1dc7640 1 -- start start 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadc1dc7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fadb4033910 0x7fadb4053d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadc1dc7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 0x7fadb404f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadc1dc7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb4052ce0 con 0x7fadb4101000 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadc1dc7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb40341b0 con 0x7fadb4033910 2024-03-08T08:29:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadc1dc7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb4038340 con 0x7fadb4054250 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.219+0000 7fadbaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fadb4033910 0x7fadb4053d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbbfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 0x7fadb404f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbbfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 0x7fadb404f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53542/0 (socket says 172.21.15.12:53542) 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbbfff640 1 -- 172.21.15.12:0/2795238959 learned_addr learned my addr 172.21.15.12:0/2795238959 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbaffd640 1 -- 172.21.15.12:0/2795238959 >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 msgr2=0x7fadb404f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbaffd640 1 --2- 172.21.15.12:0/2795238959 >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 0x7fadb404f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbaffd640 1 -- 172.21.15.12:0/2795238959 >> v1:172.21.15.156:6789/0 conn(0x7fadb4101000 legacy=0x7fadb4037c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.220+0000 7fadbaffd640 1 -- 172.21.15.12:0/2795238959 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4052f50 con 0x7fadb4033910 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.222+0000 7fadbaffd640 1 --2- 172.21.15.12:0/2795238959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4033910 0x7fadb4053d10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadb4034440 tx=0x7fadb0034620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fad51daf8fe751a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.222+0000 7fadba7fc640 1 -- 172.21.15.12:0/2795238959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb003b600 con 0x7fadb4033910 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.222+0000 7fadba7fc640 1 -- 172.21.15.12:0/2795238959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb003b7a0 con 0x7fadb4033910 2024-03-08T08:29:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.222+0000 7fadba7fc640 1 -- 172.21.15.12:0/2795238959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb003bb70 con 0x7fadb4033910 2024-03-08T08:29:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 -- 172.21.15.12:0/2795238959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4033910 msgr2=0x7fadb4053d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/2795238959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4033910 0x7fadb4053d10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadb4034440 tx=0x7fadb0034620 comp rx=0 tx=0).stop 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 -- 172.21.15.12:0/2795238959 shutdown_connections 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/2795238959 >> v2:172.21.15.156:3300/0 conn(0x7fadb4054250 0x7fadb404f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/2795238959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4033910 0x7fadb4053d10 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadb4034440 tx=0x7fadb0034620 comp rx=0 tx=0).stop 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.223+0000 7fadc1dc7640 1 -- 172.21.15.12:0/2795238959 >> 172.21.15.12:0/2795238959 conn(0x7fadb4009fa0 msgr2=0x7fadb400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.224+0000 7fadc1dc7640 1 -- 172.21.15.12:0/2795238959 shutdown_connections 2024-03-08T08:29:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.224+0000 7fadc1dc7640 1 -- 172.21.15.12:0/2795238959 wait complete. 2024-03-08T08:29:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.224+0000 7f30d9f01640 1 Processor -- start 2024-03-08T08:29:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.224+0000 7f30d9f01640 1 -- start start 2024-03-08T08:29:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.225+0000 7f096bd43640 1 Processor -- start 2024-03-08T08:29:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.225+0000 7f096bd43640 1 -- start start 2024-03-08T08:29:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d9f01640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.225+0000 7fadc1dc7640 1 Processor -- start 2024-03-08T08:29:17.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.225+0000 7fadc1dc7640 1 -- start start 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d9f01640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 0x7f30cc0df5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d9f01640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc053310 con 0x7f30cc0dfae0 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d9f01640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc0535f0 con 0x7f30cc054690 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d9f01640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0538d0 con 0x7f30cc0df1c0 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40578/0 (socket says 172.21.15.12:40578) 2024-03-08T08:29:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7fadc1dc7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 -- 172.21.15.12:0/1506952092 learned_addr learned my addr 172.21.15.12:0/1506952092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadc1dc7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadb4101000 0x7fadb402c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d2ffd640 1 --2- 172.21.15.12:0/1506952092 >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 0x7f30cc0df5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadc1dc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 0x7fadb402b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadc1dc7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb40537f0 con 0x7fadb40fd0a0 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f096bd43640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f09640d0cf0 0x7f0964165c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f096bd43640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 0x7f096406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f096bd43640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096406cfc0 con 0x7f0964162090 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f096bd43640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09640b6fe0 con 0x7f096406c670 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f096bd43640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0964166190 con 0x7f09640d0cf0 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f09692de640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f09640d0cf0 0x7f0964165c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f0969adf640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0964162090 0x7f09640b68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52902/0 (socket says 172.21.15.12:52902) 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.226+0000 7f0969adf640 1 -- 172.21.15.12:0/741304667 learned_addr learned my addr 172.21.15.12:0/741304667 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f09692de640 1 -- 172.21.15.12:0/741304667 >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 msgr2=0x7f096406ca80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T08:29:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 -- 172.21.15.12:0/1506952092 >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 msgr2=0x7f30cc0df5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1506952092 >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 0x7f30cc0df5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 -- 172.21.15.12:0/1506952092 >> v1:172.21.15.12:6789/0 conn(0x7f30cc0dfae0 legacy=0x7f30cc0e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f30d37fe640 1 -- 172.21.15.12:0/1506952092 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc053bb0 con 0x7f30cc0df1c0 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadc1dc7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4053510 con 0x7fadb4101000 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadc1dc7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4054680 con 0x7fadb4054250 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadbb7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadbaffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadb4101000 0x7fadb402c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadbb7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40586/0 (socket says 172.21.15.12:40586) 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7fadbb7fe640 1 -- 172.21.15.12:0/1918184893 learned_addr learned my addr 172.21.15.12:0/1918184893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7fadbbfff640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 0x7fadb402b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7fadbb7fe640 1 -- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadb4101000 msgr2=0x7fadb402c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7f30d2ffd640 1 --2- 172.21.15.12:0/1506952092 >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 0x7f30cc0df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:17.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f096a2e0640 1 --2- 172.21.15.12:0/741304667 >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 0x7f096406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f09692de640 1 --2- 172.21.15.12:0/741304667 >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 0x7f096406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f09692de640 1 -- 172.21.15.12:0/741304667 >> v1:172.21.15.12:6789/0 conn(0x7f0964162090 legacy=0x7f09640b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.227+0000 7f09692de640 1 -- 172.21.15.12:0/741304667 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09641663f0 con 0x7f09640d0cf0 2024-03-08T08:29:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7fadbb7fe640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadb4101000 0x7fadb402c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7fadbb7fe640 1 -- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 msgr2=0x7fadb402b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7f096a2e0640 1 --2- 172.21.15.12:0/741304667 >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 0x7f096406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:17.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7f09692de640 1 --2- 172.21.15.12:0/741304667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09640d0cf0 0x7f0964165c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09640d06b0 tx=0x7f095c02bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b446eac266c0a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1506952092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98178155fe6195 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.228+0000 7fadbb7fe640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 0x7fadb402b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fadbb7fe640 1 -- 172.21.15.12:0/1918184893 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb404cd00 con 0x7fadb4054250 2024-03-08T08:29:17.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7f0968add640 1 -- 172.21.15.12:0/741304667 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c039070 con 0x7f09640d0cf0 2024-03-08T08:29:17.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fadbbfff640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 0x7fadb402b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:17.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7f0968add640 1 -- 172.21.15.12:0/741304667 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c02ca50 con 0x7f09640d0cf0 2024-03-08T08:29:17.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7f30d27fc640 1 -- 172.21.15.12:0/1506952092 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8034070 con 0x7f30cc0df1c0 2024-03-08T08:29:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fadbb7fe640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fada800b3e0 tx=0x7fada800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8015020 con 0x7fadb4054250 2024-03-08T08:29:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb404cfe0 con 0x7fadb4054250 2024-03-08T08:29:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada8011850 con 0x7fadb4054250 2024-03-08T08:29:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb404dfe0 con 0x7fadb4054250 2024-03-08T08:29:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.229+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8011c40 con 0x7fadb4054250 2024-03-08T08:29:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.230+0000 7f30d27fc640 1 -- 172.21.15.12:0/1506952092 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c8030880 con 0x7f30cc0df1c0 2024-03-08T08:29:17.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.230+0000 7f096bd43640 1 -- 172.21.15.12:0/741304667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09640d0cf0 msgr2=0x7f0964165c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.230+0000 7f096bd43640 1 --2- 172.21.15.12:0/741304667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09640d0cf0 0x7f0964165c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09640d06b0 tx=0x7f095c02bd20 comp rx=0 tx=0).stop 2024-03-08T08:29:17.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.230+0000 7f30d27fc640 1 -- 172.21.15.12:0/1506952092 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8030c90 con 0x7f30cc0df1c0 2024-03-08T08:29:17.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 -- 172.21.15.12:0/741304667 shutdown_connections 2024-03-08T08:29:17.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 --2- 172.21.15.12:0/741304667 >> v2:172.21.15.12:3300/0 conn(0x7f096406c670 0x7f096406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 --2- 172.21.15.12:0/741304667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09640d0cf0 0x7f0964165c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 -- 172.21.15.12:0/741304667 >> 172.21.15.12:0/741304667 conn(0x7f0964072720 msgr2=0x7f0964072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 -- 172.21.15.12:0/1506952092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 msgr2=0x7f30cc04fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1506952092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b9c0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 -- 172.21.15.12:0/741304667 shutdown_connections 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f096bd43640 1 -- 172.21.15.12:0/741304667 wait complete. 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 -- 172.21.15.12:0/1506952092 shutdown_connections 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1506952092 >> v2:172.21.15.156:3300/0 conn(0x7f30cc054690 0x7f30cc0df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1506952092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0df1c0 0x7f30cc04fff0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 -- 172.21.15.12:0/1506952092 >> 172.21.15.12:0/1506952092 conn(0x7f30cc009fe0 msgr2=0x7f30cc00a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 -- 172.21.15.12:0/1506952092 shutdown_connections 2024-03-08T08:29:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.231+0000 7f30d9f01640 1 -- 172.21.15.12:0/1506952092 wait complete. 2024-03-08T08:29:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.233+0000 7f30d9f01640 1 Processor -- start 2024-03-08T08:29:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.233+0000 7f096bd43640 1 Processor -- start 2024-03-08T08:29:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.233+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fada8018020 con 0x7fadb4054250 2024-03-08T08:29:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.233+0000 7f096bd43640 1 -- start start 2024-03-08T08:29:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.233+0000 7f30d9f01640 1 -- start start 2024-03-08T08:29:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7fada7fff640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 0x7fada006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c005290 con 0x7fadb4054250 2024-03-08T08:29:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fada808a090 con 0x7fadb4054250 2024-03-08T08:29:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7fadbaffd640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 0x7fada006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7f096bd43640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f096406c670 0x7f0964160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7f096bd43640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09640d0cf0 0x7f09640c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7f30d9f01640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 0x7f30cc0d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7f096bd43640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096bd43640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09640d1a10 con 0x7f0964162090 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096bd43640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09641669b0 con 0x7f096406c670 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096bd43640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0964162470 con 0x7f09640d0cf0 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.234+0000 7f30d9f01640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 0x7f30cc0d9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d9f01640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0dfae0 0x7f30cc0d7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d9f01640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc050530 con 0x7f30cc0df1c0 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d9f01640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc054170 con 0x7f30cc054690 2024-03-08T08:29:17.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d9f01640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0e04e0 con 0x7f30cc0dfae0 2024-03-08T08:29:17.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 0x7f30cc0d9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58098/0 (socket says 172.21.15.12:58098) 2024-03-08T08:29:17.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 -- 172.21.15.12:0/900089818 learned_addr learned my addr 172.21.15.12:0/900089818 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 0x7f30cc0d9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58086/0 (socket says 172.21.15.12:58086) 2024-03-08T08:29:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f09692de640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09640d0cf0 0x7f09640c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d2ffd640 1 -- 172.21.15.12:0/1018996928 learned_addr learned my addr 172.21.15.12:0/1018996928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7fadbaffd640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 0x7fada006da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadb003e000 tx=0x7fadb0039000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 0x7f30cc0d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f0969adf640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f096406c670 0x7f0964160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 -- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09640d0cf0 msgr2=0x7f09640c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09640d0cf0 0x7f09640c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 -- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f096406c670 msgr2=0x7f0964160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f096406c670 0x7f0964160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f096a2e0640 1 -- 172.21.15.12:0/900089818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09641ed450 con 0x7f0964162090 2024-03-08T08:29:17.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.235+0000 7f30d3fff640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0dfae0 0x7f30cc0d7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 -- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0dfae0 msgr2=0x7f30cc0d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0dfae0 0x7f30cc0d7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 -- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 msgr2=0x7f30cc0d9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 0x7f30cc0d9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 -- 172.21.15.12:0/1018996928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0d8850 con 0x7f30cc054690 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f30d37fe640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 0x7f30cc0d64d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f30cc053a50 tx=0x7f30c8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f096a2e0640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f095400c820 tx=0x7f095400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954015070 con 0x7f0964162090 2024-03-08T08:29:17.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.236+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0964160940 con 0x7f0964162090 2024-03-08T08:29:17.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8034040 con 0x7f30cc054690 2024-03-08T08:29:17.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09641ecb20 con 0x7f0964162090 2024-03-08T08:29:17.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc0d7fb0 con 0x7f30cc054690 2024-03-08T08:29:17.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0954007a30 con 0x7f0964162090 2024-03-08T08:29:17.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc0d8570 con 0x7f30cc054690 2024-03-08T08:29:17.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.238+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954007e20 con 0x7f0964162090 2024-03-08T08:29:17.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.237+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c8039c10 con 0x7f30cc054690 2024-03-08T08:29:17.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.238+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c803f4c0 con 0x7f30cc054690 2024-03-08T08:29:17.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.239+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0954018070 con 0x7f0964162090 2024-03-08T08:29:17.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.239+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b4005290 con 0x7f30cc054690 2024-03-08T08:29:17.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f095a7fc640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 0x7f094806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30c806a020 con 0x7f30cc054690 2024-03-08T08:29:17.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fada801f020 con 0x7fadb4054250 2024-03-08T08:29:17.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f0969adf640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 0x7f094806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f30bffff640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 0x7f30b806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30c80b07e0 con 0x7f30cc054690 2024-03-08T08:29:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.240+0000 7f30d2ffd640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 0x7f30b806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.241+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0954089ff0 con 0x7f0964162090 2024-03-08T08:29:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.241+0000 7f0969adf640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 0x7f094806da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0960007fd0 tx=0x7f0960007c20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.241+0000 7f30d2ffd640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 0x7f30b806da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30cc17a160 tx=0x7f30c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.243+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c005290 con 0x7f0964162090 2024-03-08T08:29:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.245+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f30c807f6d0 con 0x7f30cc054690 2024-03-08T08:29:17.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.252+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f09540588c0 con 0x7f0964162090 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.288+0000 7f99de0e5640 1 Processor -- start 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.288+0000 7f99de0e5640 1 -- start start 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.289+0000 7f99de0e5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99d80b3280 0x7f99d816a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.289+0000 7f99de0e5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 0x7f99d8071b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99de0e5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d816e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99de0e5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d8172510 con 0x7f99d80b3280 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99de0e5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d80b3660 con 0x7f99d8071690 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99de0e5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d816b220 con 0x7f99d816e1f0 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99d80b3280 0x7f99d816a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99d80b3280 0x7f99d816a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58106/0 (socket says 172.21.15.12:58106) 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 -- 172.21.15.12:0/1050099224 learned_addr learned my addr 172.21.15.12:0/1050099224 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.289+0000 7fd518181640 1 Processor -- start 2024-03-08T08:29:17.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.289+0000 7fd518181640 1 -- start start 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dc8e2640 1 --2- 172.21.15.12:0/1050099224 >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 0x7f99d8071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd518181640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd5100cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd518181640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd51009a980 0x7fd5100b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd518181640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5100cd820 con 0x7fd5100cfa00 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd518181640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51009a4f0 con 0x7fd51009a980 2024-03-08T08:29:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd518181640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd510099bc0 con 0x7fd5100d04f0 2024-03-08T08:29:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.291+0000 7f8549ec0640 1 Processor -- start 2024-03-08T08:29:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.291+0000 7f8549ec0640 1 -- start start 2024-03-08T08:29:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd515f1d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd5100cfa00 0x7fd5100cfde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52910/0 (socket says 172.21.15.12:52910) 2024-03-08T08:29:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd515f1d640 1 -- 172.21.15.12:0/2528241696 learned_addr learned my addr 172.21.15.12:0/2528241696 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7fd51671e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd51009a980 0x7fd5100b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 -- 172.21.15.12:0/1050099224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 msgr2=0x7f99d816e680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T08:29:17.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7fd51671e640 1 -- 172.21.15.12:0/2528241696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 msgr2=0x7fd5100cc490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T08:29:17.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7fd51671e640 1 --2- 172.21.15.12:0/2528241696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd5100cc490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7fd51671e640 1 -- 172.21.15.12:0/2528241696 >> v1:172.21.15.12:6789/0 conn(0x7fd5100cfa00 legacy=0x7fd5100cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7fd51671e640 1 -- 172.21.15.12:0/2528241696 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510099ea0 con 0x7fd51009a980 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f854415de90 0x7f85440d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 0x7f854406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f854406cbd0 0x7f85440d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854406d030 con 0x7f854406cbd0 2024-03-08T08:29:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854415e270 con 0x7f85440d20d0 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8549ec0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440d2650 con 0x7f854415de90 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8542ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 0x7f854406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8543fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f854406cbd0 0x7f85440d3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8542ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 0x7f854406c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53604/0 (socket says 172.21.15.12:53604) 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7f8542ffd640 1 -- 172.21.15.12:0/1358038960 learned_addr learned my addr 172.21.15.12:0/1358038960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7ff2390df640 1 Processor -- start 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 --2- 172.21.15.12:0/1050099224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d816e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 -- 172.21.15.12:0/1050099224 >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 msgr2=0x7f99d8071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 --2- 172.21.15.12:0/1050099224 >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 0x7f99d8071b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dd0e3640 1 -- 172.21.15.12:0/1050099224 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d816b500 con 0x7f99d80b3280 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.290+0000 7f99dc8e2640 1 --2- 172.21.15.12:0/1050099224 >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 0x7f99d8071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.292+0000 7ff2390df640 1 -- start start 2024-03-08T08:29:17.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f8543fff640 1 -- 172.21.15.12:0/1358038960 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f854415de90 msgr2=0x7f85440d1b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f8543fff640 1 --2- 172.21.15.12:0/1358038960 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f854415de90 0x7f85440d1b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f8543fff640 1 -- 172.21.15.12:0/1358038960 >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 msgr2=0x7f854406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f8543fff640 1 --2- 172.21.15.12:0/1358038960 >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 0x7f854406c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f8543fff640 1 -- 172.21.15.12:0/1358038960 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440d2930 con 0x7f854406cbd0 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f99dd0e3640 1 --2- 172.21.15.12:0/1050099224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d80b3280 0x7f99d816a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009870 tx=0x7f99cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6354c7b298b3bf3b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7fd51671e640 1 --2- 172.21.15.12:0/2528241696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100b12e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097a0 tx=0x7fd50c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85af9fd78391cf77 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7ff2390df640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff23415fa90 0x7ff2340714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7ff2390df640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff234071a20 0x7ff2340b4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7ff2390df640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff234163570 con 0x7ff2340b4a40 2024-03-08T08:29:17.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7ff2390df640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff234071fa0 con 0x7ff234071a20 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7ff2390df640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2341627b0 con 0x7ff23415fa90 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff23359e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff2340b4a40 0x7ff23406d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52914/0 (socket says 172.21.15.12:52914) 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff23359e640 1 -- 172.21.15.12:0/3656444738 learned_addr learned my addr 172.21.15.12:0/3656444738 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff232d9d640 1 --2- 172.21.15.12:0/3656444738 >> v2:172.21.15.156:3300/0 conn(0x7ff23415fa90 0x7ff2340714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f8543fff640 1 --2- 172.21.15.12:0/1358038960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440d3a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85440d4ae0 tx=0x7f853c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fc4e77c1166feb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f99d7fff640 1 -- 172.21.15.12:0/1050099224 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc035020 con 0x7f99d80b3280 2024-03-08T08:29:17.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f99d7fff640 1 -- 172.21.15.12:0/1050099224 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc031870 con 0x7f99d80b3280 2024-03-08T08:29:17.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.293+0000 7f99d7fff640 1 -- 172.21.15.12:0/1050099224 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc031c40 con 0x7f99d80b3280 2024-03-08T08:29:17.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f85427fc640 1 -- 172.21.15.12:0/1358038960 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c039070 con 0x7f854406cbd0 2024-03-08T08:29:17.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff231d9b640 1 -- 172.21.15.12:0/3656444738 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224275641 0 0) 0x7ff234163570 con 0x7ff2340b4a40 2024-03-08T08:29:17.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff231d9b640 1 -- 172.21.15.12:0/3656444738 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff21c003520 con 0x7ff2340b4a40 2024-03-08T08:29:17.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff232d9d640 1 -- 172.21.15.12:0/3656444738 >> v2:172.21.15.12:3300/0 conn(0x7ff234071a20 msgr2=0x7ff2340b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f85427fc640 1 -- 172.21.15.12:0/1358038960 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c02cc90 con 0x7f854406cbd0 2024-03-08T08:29:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f85427fc640 1 -- 172.21.15.12:0/1358038960 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c034660 con 0x7f854406cbd0 2024-03-08T08:29:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7fd514f1b640 1 -- 172.21.15.12:0/2528241696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0315a0 con 0x7fd51009a980 2024-03-08T08:29:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff232d9d640 1 --2- 172.21.15.12:0/3656444738 >> v2:172.21.15.12:3300/0 conn(0x7ff234071a20 0x7ff2340b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7ff232d9d640 1 -- 172.21.15.12:0/3656444738 >> v1:172.21.15.12:6789/0 conn(0x7ff2340b4a40 legacy=0x7ff23406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7fd514f1b640 1 -- 172.21.15.12:0/2528241696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c037070 con 0x7fd51009a980 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f99de0e5640 1 -- 172.21.15.12:0/1050099224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d80b3280 msgr2=0x7f99d816a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.294+0000 7f99de0e5640 1 --2- 172.21.15.12:0/1050099224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d80b3280 0x7f99d816a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009870 tx=0x7f99cc02b910 comp rx=0 tx=0).stop 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f99de0e5640 1 -- 172.21.15.12:0/1050099224 shutdown_connections 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f99de0e5640 1 --2- 172.21.15.12:0/1050099224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d816e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f99de0e5640 1 --2- 172.21.15.12:0/1050099224 >> v2:172.21.15.156:3300/0 conn(0x7f99d8071690 0x7f99d8071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f99de0e5640 1 --2- 172.21.15.12:0/1050099224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d80b3280 0x7f99d816a870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f8549ec0640 1 -- 172.21.15.12:0/1358038960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 msgr2=0x7f85440d3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f8549ec0640 1 --2- 172.21.15.12:0/1358038960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440d3a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85440d4ae0 tx=0x7f853c02bee0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 -- 172.21.15.12:0/2528241696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 msgr2=0x7fd5100b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 -- 172.21.15.12:0/1358038960 shutdown_connections 2024-03-08T08:29:17.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 --2- 172.21.15.12:0/1358038960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440d3a10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 --2- 172.21.15.12:0/1358038960 >> v2:172.21.15.156:3300/0 conn(0x7f85440d20d0 0x7f854406c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 --2- 172.21.15.12:0/1358038960 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f854415de90 0x7f85440d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 -- 172.21.15.12:0/1358038960 >> 172.21.15.12:0/1358038960 conn(0x7f8544072720 msgr2=0x7f8544072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 --2- 172.21.15.12:0/2528241696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100b12e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097a0 tx=0x7fd50c02b840 comp rx=0 tx=0).stop 2024-03-08T08:29:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7f99de0e5640 1 -- 172.21.15.12:0/1050099224 >> 172.21.15.12:0/1050099224 conn(0x7f99d806aff0 msgr2=0x7f99d806b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 -- 172.21.15.12:0/2528241696 shutdown_connections 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 --2- 172.21.15.12:0/2528241696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100b12e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 --2- 172.21.15.12:0/2528241696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd5100cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 -- 172.21.15.12:0/2528241696 >> 172.21.15.12:0/2528241696 conn(0x7fd5100b3470 msgr2=0x7fd5100b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 -- 172.21.15.12:0/1358038960 shutdown_connections 2024-03-08T08:29:17.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f8549ec0640 1 -- 172.21.15.12:0/1358038960 wait complete. 2024-03-08T08:29:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f99de0e5640 1 -- 172.21.15.12:0/1050099224 shutdown_connections 2024-03-08T08:29:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.295+0000 7ff232d9d640 1 -- 172.21.15.12:0/3656444738 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234162a90 con 0x7ff23415fa90 2024-03-08T08:29:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7f99de0e5640 1 -- 172.21.15.12:0/1050099224 wait complete. 2024-03-08T08:29:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7ff232d9d640 1 --2- 172.21.15.12:0/3656444738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff23415fa90 0x7ff2340714e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2340b5030 tx=0x7ff22c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d61582b4641d91c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 -- 172.21.15.12:0/2528241696 shutdown_connections 2024-03-08T08:29:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7ff231d9b640 1 -- 172.21.15.12:0/3656444738 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c039070 con 0x7ff23415fa90 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7ff231d9b640 1 -- 172.21.15.12:0/3656444738 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c02cc80 con 0x7ff23415fa90 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7ff231d9b640 1 -- 172.21.15.12:0/3656444738 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c034660 con 0x7ff23415fa90 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 -- 172.21.15.12:0/3656444738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff23415fa90 msgr2=0x7ff2340714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 --2- 172.21.15.12:0/3656444738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff23415fa90 0x7ff2340714e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2340b5030 tx=0x7ff22c02bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 -- 172.21.15.12:0/3656444738 shutdown_connections 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 --2- 172.21.15.12:0/3656444738 >> v2:172.21.15.12:3300/0 conn(0x7ff234071a20 0x7ff2340b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 --2- 172.21.15.12:0/3656444738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff23415fa90 0x7ff2340714e0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2340b5030 tx=0x7ff22c02bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.297+0000 7ff2390df640 1 -- 172.21.15.12:0/3656444738 >> 172.21.15.12:0/3656444738 conn(0x7ff23406abd0 msgr2=0x7ff23406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.296+0000 7fd518181640 1 -- 172.21.15.12:0/2528241696 wait complete. 2024-03-08T08:29:17.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7f8549ec0640 1 Processor -- start 2024-03-08T08:29:17.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7f99de0e5640 1 Processor -- start 2024-03-08T08:29:17.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7f8549ec0640 1 -- start start 2024-03-08T08:29:17.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7fd518181640 1 Processor -- start 2024-03-08T08:29:17.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7f8549ec0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7fd518181640 1 -- start start 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5100cfa00 0x7fd5100c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd510204730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100cd480 con 0x7fd51009a980 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51009ae10 con 0x7fd5100cfa00 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7fd518181640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100cd600 con 0x7fd5100d04f0 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58166/0 (socket says 172.21.15.12:58166) 2024-03-08T08:29:17.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 -- 172.21.15.12:0/3351158955 learned_addr learned my addr 172.21.15.12:0/3351158955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8549ec0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 0x7f8544159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f85437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd51671e640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd510204730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd51571c640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5100cfa00 0x7fd5100c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8549ec0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854415de90 0x7f85441606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8542ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 0x7f8544159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f85437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58176/0 (socket says 172.21.15.12:58176) 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8542ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 0x7f8544159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40648/0 (socket says 172.21.15.12:40648) 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f85437fe640 1 -- 172.21.15.12:0/3034388384 learned_addr learned my addr 172.21.15.12:0/3034388384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440d31d0 con 0x7f854406cbd0 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 -- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 msgr2=0x7fd510204730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd510204730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 -- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5100cfa00 msgr2=0x7fd5100c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5100cfa00 0x7fd5100c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7fd515f1d640 1 -- 172.21.15.12:0/3351158955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5100c9930 con 0x7fd51009a980 2024-03-08T08:29:17.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7ff2390df640 1 -- 172.21.15.12:0/3656444738 shutdown_connections 2024-03-08T08:29:17.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7ff2390df640 1 -- 172.21.15.12:0/3656444738 wait complete. 2024-03-08T08:29:17.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7ff2390df640 1 Processor -- start 2024-03-08T08:29:17.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7ff2390df640 1 -- start start 2024-03-08T08:29:17.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440d2ef0 con 0x7f854415de90 2024-03-08T08:29:17.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.300+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440d36f0 con 0x7f85440d20d0 2024-03-08T08:29:17.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f8543fff640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854415de90 0x7f85441606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7fd515f1d640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd50000c480 tx=0x7fd50000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 -- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 msgr2=0x7f8544159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 0x7f8544159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 -- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854415de90 msgr2=0x7f85441606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854415de90 0x7f85441606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440c86a0 con 0x7f854406cbd0 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500005ce0 con 0x7fd51009a980 2024-03-08T08:29:17.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.298+0000 7f99de0e5640 1 -- start start 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7f99de0e5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d8071690 0x7f99d80a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7f99de0e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99d80b3280 0x7f99d80b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.299+0000 7f99dd0e3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d8071690 0x7f99d80a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f99de0e5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f99de0e5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d816ebc0 con 0x7f99d8071690 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f99de0e5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d816bd00 con 0x7f99d80b3280 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f99de0e5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d816ed40 con 0x7f99d816e1f0 2024-03-08T08:29:17.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7f85437fe640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f853400b480 tx=0x7f853400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd510202210 con 0x7fd51009a980 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.301+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5100c9eb0 con 0x7fd51009a980 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd500007070 con 0x7fd51009a980 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5000074e0 con 0x7fd51009a980 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534014070 con 0x7f854406cbd0 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534010890 con 0x7f854406cbd0 2024-03-08T08:29:17.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534010c80 con 0x7f854406cbd0 2024-03-08T08:29:17.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2340b4a40 0x7ff234160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff23408a730 0x7ff234160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340b3950 con 0x7ff234071a20 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff234163050 con 0x7ff2340b4a40 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff2390df640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff234166a30 con 0x7ff23408a730 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff23259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2340b4a40 0x7ff234160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff232d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff232d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58178/0 (socket says 172.21.15.12:58178) 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.302+0000 7ff232d9d640 1 -- 172.21.15.12:0/71221961 learned_addr learned my addr 172.21.15.12:0/71221961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 -- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff23408a730 msgr2=0x7ff234160950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff23408a730 0x7ff234160950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 -- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2340b4a40 msgr2=0x7ff234160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2340b4a40 0x7ff234160410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 -- 172.21.15.12:0/71221961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340a9000 con 0x7ff234071a20 2024-03-08T08:29:17.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85440c8980 con 0x7f854406cbd0 2024-03-08T08:29:17.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85440c8f40 con 0x7f854406cbd0 2024-03-08T08:29:17.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.303+0000 7ff232d9d640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff22c000f80 tx=0x7ff22c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8005290 con 0x7fd51009a980 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c044020 con 0x7ff234071a20 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2340a92e0 con 0x7ff234071a20 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2342084d0 con 0x7ff234071a20 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c03a040 con 0x7ff234071a20 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c039070 con 0x7ff234071a20 2024-03-08T08:29:17.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7f99dd8e4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7f99dd8e4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40656/0 (socket says 172.21.15.12:40656) 2024-03-08T08:29:17.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7f99dd8e4640 1 -- 172.21.15.12:0/3670581620 learned_addr learned my addr 172.21.15.12:0/3670581620 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.304+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85440749c0 con 0x7f854406cbd0 2024-03-08T08:29:17.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dc8e2640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99d80b3280 0x7f99d80b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dd8e4640 1 -- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99d80b3280 msgr2=0x7f99d80b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dd8e4640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99d80b3280 0x7f99d80b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dd8e4640 1 -- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d8071690 msgr2=0x7f99d80a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dd8e4640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d8071690 0x7f99d80a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.305+0000 7f99dd8e4640 1 -- 172.21.15.12:0/3670581620 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80acb30 con 0x7f99d816e1f0 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7f99dd8e4640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99c800b990 tx=0x7f99c800be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd50001b400 con 0x7fd51009a980 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8019070 con 0x7f99d816e1f0 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d80a89e0 con 0x7f99d816e1f0 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8534017070 con 0x7f854406cbd0 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f852bfff640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 0x7f853006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f853408a460 con 0x7f854406cbd0 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800cba0 con 0x7f99d816e1f0 2024-03-08T08:29:17.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d80acdb0 con 0x7f99d816e1f0 2024-03-08T08:29:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8019070 con 0x7f99d816e1f0 2024-03-08T08:29:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.307+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff22c047020 con 0x7ff234071a20 2024-03-08T08:29:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7fd5067fc640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 0x7fd4f406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.306+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd500089cb0 con 0x7fd51009a980 2024-03-08T08:29:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.308+0000 7fd51571c640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 0x7fd4f406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.308+0000 7ff21b7fe640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 0x7ff22006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.308+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff214005290 con 0x7ff234071a20 2024-03-08T08:29:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.309+0000 7ff23259c640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 0x7ff22006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.309+0000 7f8542ffd640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 0x7f853006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.309+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d80749c0 con 0x7f99d816e1f0 2024-03-08T08:29:17.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.309+0000 7fd51571c640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 0x7fd4f406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd508009fd0 tx=0x7fd50800d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.310+0000 7f8542ffd640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 0x7f853006da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f85440c7d30 tx=0x7f8538008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.310+0000 7ff23259c640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 0x7ff22006da10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2340cc160 tx=0x7ff228008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.310+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99c801c070 con 0x7f99d816e1f0 2024-03-08T08:29:17.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.310+0000 7f99d5ffb640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 0x7f99c406dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.311+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99c800c9a0 con 0x7f99d816e1f0 2024-03-08T08:29:17.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.311+0000 7f99dd0e3640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 0x7f99c406dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.311+0000 7f99dd0e3640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 0x7f99c406dd90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f99cc036000 tx=0x7f99cc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.313+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff22c079330 con 0x7ff234071a20 2024-03-08T08:29:17.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.314+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f853401e030 con 0x7f854406cbd0 2024-03-08T08:29:17.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.315+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f99d80acb30 con 0x7f99d816e1f0 2024-03-08T08:29:17.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.322+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd500058550 con 0x7fd51009a980 2024-03-08T08:29:17.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.330+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff22c04e030 con 0x7ff234071a20 2024-03-08T08:29:17.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.572+0000 7f0567fff640 1 -- 172.21.15.12:0/664874018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0548005580 con 0x7f056003aa10 2024-03-08T08:29:17.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.572+0000 7f054ffff640 1 -- 172.21.15.12:0/664874018 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05680a8040 con 0x7f056003aa10 2024-03-08T08:29:17.574 INFO:teuthology.orchestra.run.smithi012.stdout:60129542149 2024-03-08T08:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 msgr2=0x7f055006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 0x7f055006dba0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0558004fb0 tx=0x7f0558007e30 comp rx=0 tx=0).stop 2024-03-08T08:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 msgr2=0x7f0560050290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f05680508c0 tx=0x7f0568062a30 comp rx=0 tx=0).stop 2024-03-08T08:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 shutdown_connections 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f055006b6e0 0x7f055006dba0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0560048c30 0x7f0560033a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.576+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f056003aa10 0x7f0560050290 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.577+0000 7f054dffb640 1 --2- 172.21.15.12:0/664874018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0560009180 0x7f0560185640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.577+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 >> 172.21.15.12:0/664874018 conn(0x7f056001ef30 msgr2=0x7f05600eb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.577+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 shutdown_connections 2024-03-08T08:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.577+0000 7f054dffb640 1 -- 172.21.15.12:0/664874018 wait complete. 2024-03-08T08:29:17.595 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2024-03-08T08:29:17.595 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.648+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd4f8005580 con 0x7fd51009a980 2024-03-08T08:29:17.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.649+0000 7fd5067fc640 1 -- 172.21.15.12:0/3351158955 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd50005c5f0 con 0x7fd51009a980 2024-03-08T08:29:17.650 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:17.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 msgr2=0x7fd4f406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 0x7fd4f406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd508009fd0 tx=0x7fd50800d040 comp rx=0 tx=0).stop 2024-03-08T08:29:17.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fad9c005580 con 0x7fadb4054250 2024-03-08T08:29:17.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 msgr2=0x7fd5100cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd50000c480 tx=0x7fd50000c940 comp rx=0 tx=0).stop 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 shutdown_connections 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd4f406b520 0x7fd4f406d9e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5100d04f0 0x7fd510204730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5100cfa00 0x7fd5100c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 --2- 172.21.15.12:0/3351158955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51009a980 0x7fd5100cb690 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.651+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 >> 172.21.15.12:0/3351158955 conn(0x7fd5100b3470 msgr2=0x7fd51006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.652+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 shutdown_connections 2024-03-08T08:29:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.652+0000 7fd518181640 1 -- 172.21.15.12:0/3351158955 wait complete. 2024-03-08T08:29:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.652+0000 7fada7fff640 1 -- 172.21.15.12:0/1918184893 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fada80172c0 con 0x7fadb4054250 2024-03-08T08:29:17.655 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 msgr2=0x7fada006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 0x7fada006da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadb003e000 tx=0x7fadb0039000 comp rx=0 tx=0).stop 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 msgr2=0x7fadb404c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fada800b3e0 tx=0x7fada800b8a0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 shutdown_connections 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fada006b5a0 0x7fada006da60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40fd0a0 0x7fadb402b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadb4101000 0x7fadb402c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 --2- 172.21.15.12:0/1918184893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fadb4054250 0x7fadb404c590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 >> 172.21.15.12:0/1918184893 conn(0x7fadb4009fa0 msgr2=0x7fadb4036520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 shutdown_connections 2024-03-08T08:29:17.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.655+0000 7fadc1dc7640 1 -- 172.21.15.12:0/1918184893 wait complete. 2024-03-08T08:29:17.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-03-08T08:29:17.669 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-03-08T08:29:17.671 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.673+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f30b4005580 con 0x7f30cc054690 2024-03-08T08:29:17.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.673+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f094c005580 con 0x7f0964162090 2024-03-08T08:29:17.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.673+0000 7f30bffff640 1 -- 172.21.15.12:0/1018996928 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f30c807efd0 con 0x7f30cc054690 2024-03-08T08:29:17.675 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:17.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.673+0000 7f095a7fc640 1 -- 172.21.15.12:0/900089818 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f095405c960 con 0x7f0964162090 2024-03-08T08:29:17.675 INFO:teuthology.orchestra.run.smithi012.stdout:64424509448 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.675+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 msgr2=0x7f30b806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.675+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 0x7f30b806da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30cc17a160 tx=0x7f30c0008040 comp rx=0 tx=0).stop 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.675+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 msgr2=0x7f30cc0d64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.675+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 0x7f30cc0d64d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f30cc053a50 tx=0x7f30c8004730 comp rx=0 tx=0).stop 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 msgr2=0x7f094806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 0x7f094806da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0960007fd0 tx=0x7f0960007c20 comp rx=0 tx=0).stop 2024-03-08T08:29:17.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 msgr2=0x7f09640c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f095400c820 tx=0x7f095400cce0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 shutdown_connections 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f30b806b5a0 0x7f30b806da60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30cc0dfae0 0x7f30cc0d7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30cc0df1c0 0x7f30cc0d9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 --2- 172.21.15.12:0/1018996928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30cc054690 0x7f30cc0d64d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 >> 172.21.15.12:0/1018996928 conn(0x7f30cc009fe0 msgr2=0x7f30cc04ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 shutdown_connections 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f094806b5a0 0x7f094806da60 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0960007fd0 tx=0x7f0960007c20 comp rx=0 tx=0).stop 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0964162090 0x7f09640c9870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09640d0cf0 0x7f09640c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 --2- 172.21.15.12:0/900089818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f096406c670 0x7f0964160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 >> 172.21.15.12:0/900089818 conn(0x7f0964072720 msgr2=0x7f09640d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 shutdown_connections 2024-03-08T08:29:17.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f096bd43640 1 -- 172.21.15.12:0/900089818 wait complete. 2024-03-08T08:29:17.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 shutdown_connections 2024-03-08T08:29:17.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.676+0000 7f30d9f01640 1 -- 172.21.15.12:0/1018996928 wait complete. 2024-03-08T08:29:17.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.1 2024-03-08T08:29:17.689 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.690 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.3 2024-03-08T08:29:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.689+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f85440b3750 con 0x7f854406cbd0 2024-03-08T08:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.689+0000 7f852bfff640 1 -- 172.21.15.12:0/3034388384 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8534010e20 con 0x7f854406cbd0 2024-03-08T08:29:17.691 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.691 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 msgr2=0x7f853006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 0x7f853006da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f85440c7d30 tx=0x7f8538008040 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 msgr2=0x7f85440aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f853400b480 tx=0x7f853400b940 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 shutdown_connections 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f853006b5d0 0x7f853006da90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854415de90 0x7f85441606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85440d20d0 0x7f8544159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 --2- 172.21.15.12:0/3034388384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854406cbd0 0x7f85440aa4d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 >> 172.21.15.12:0/3034388384 conn(0x7f8544072720 msgr2=0x7f85440d3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 shutdown_connections 2024-03-08T08:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.691+0000 7f8549ec0640 1 -- 172.21.15.12:0/3034388384 wait complete. 2024-03-08T08:29:17.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-03-08T08:29:17.703 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.728+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f99d80b4490 con 0x7f99d816e1f0 2024-03-08T08:29:17.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.728+0000 7f99d5ffb640 1 -- 172.21.15.12:0/3670581620 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f99c8058130 con 0x7f99d816e1f0 2024-03-08T08:29:17.730 INFO:teuthology.orchestra.run.smithi012.stdout:60129542150 2024-03-08T08:29:17.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 msgr2=0x7f99c406dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 0x7f99c406dd90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f99cc036000 tx=0x7f99cc03c040 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 msgr2=0x7f99d8169530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99c800b990 tx=0x7f99c800be50 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 shutdown_connections 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f99c406b8d0 0x7f99c406dd90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99d816e1f0 0x7f99d8169530 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99d80b3280 0x7f99d80b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 --2- 172.21.15.12:0/3670581620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99d8071690 0x7f99d80a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 >> 172.21.15.12:0/3670581620 conn(0x7f99d806aff0 msgr2=0x7f99d816a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 shutdown_connections 2024-03-08T08:29:17.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.731+0000 7f99de0e5640 1 -- 172.21.15.12:0/3670581620 wait complete. 2024-03-08T08:29:17.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.737+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff214005580 con 0x7ff234071a20 2024-03-08T08:29:17.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.738+0000 7ff21b7fe640 1 -- 172.21.15.12:0/71221961 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff22c07b990 con 0x7ff234071a20 2024-03-08T08:29:17.739 INFO:teuthology.orchestra.run.smithi012.stdout:60129542151 2024-03-08T08:29:17.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 msgr2=0x7ff22006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 0x7ff22006da10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2340cc160 tx=0x7ff228008040 comp rx=0 tx=0).stop 2024-03-08T08:29:17.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 msgr2=0x7ff23415a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff22c000f80 tx=0x7ff22c004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 shutdown_connections 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff22006b550 0x7ff22006da10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff23408a730 0x7ff234160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2340b4a40 0x7ff234160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 --2- 172.21.15.12:0/71221961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff234071a20 0x7ff23415a3e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 >> 172.21.15.12:0/71221961 conn(0x7ff23406abd0 msgr2=0x7ff23415ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 shutdown_connections 2024-03-08T08:29:17.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.740+0000 7ff2390df640 1 -- 172.21.15.12:0/71221961 wait complete. 2024-03-08T08:29:17.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2024-03-08T08:29:17.743 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-03-08T08:29:17.752 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:17.752 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-08T08:29:17.752 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:17.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.892+0000 7f91277d7640 1 Processor -- start 2024-03-08T08:29:17.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.892+0000 7f91277d7640 1 -- start start 2024-03-08T08:29:17.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f91277d7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f912006c240 0x7f912006c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f91277d7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912015e930 con 0x7f91200d24b0 2024-03-08T08:29:17.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f91277d7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912006cb60 con 0x7f91200b9420 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f91277d7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91200b6b30 con 0x7f912006c240 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f912006c240 0x7f912006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f912006c240 0x7f912006c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58180/0 (socket says 172.21.15.12:58180) 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125d74640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f91200d24b0 0x7f91200d2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52922/0 (socket says 172.21.15.12:52922) 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 -- 172.21.15.12:0/22169551 learned_addr learned my addr 172.21.15.12:0/22169551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 -- 172.21.15.12:0/22169551 >> v1:172.21.15.156:6789/0 conn(0x7f91200b9420 legacy=0x7f912015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 -- 172.21.15.12:0/22169551 >> v1:172.21.15.12:6789/0 conn(0x7f91200d24b0 legacy=0x7f91200d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9125573640 1 -- 172.21.15.12:0/22169551 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912015ebc0 con 0x7f912006c240 2024-03-08T08:29:17.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.893+0000 7f9117fff640 1 -- 172.21.15.12:0/22169551 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937756810 0 0) 0x7f912015e930 con 0x7f91200d24b0 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f9125573640 1 --2- 172.21.15.12:0/22169551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f912006c240 0x7f912006c620 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f911c009870 tx=0x7f911c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1aee8fbbdcabe3c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f9117fff640 1 -- 172.21.15.12:0/22169551 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c02f840 con 0x7f912006c240 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f9117fff640 1 -- 172.21.15.12:0/22169551 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c02f9e0 con 0x7f912006c240 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f9117fff640 1 -- 172.21.15.12:0/22169551 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c02fdb0 con 0x7f912006c240 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f91277d7640 1 -- 172.21.15.12:0/22169551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f912006c240 msgr2=0x7f912006c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f91277d7640 1 --2- 172.21.15.12:0/22169551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f912006c240 0x7f912006c620 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f911c009870 tx=0x7f911c02ba20 comp rx=0 tx=0).stop 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f91277d7640 1 -- 172.21.15.12:0/22169551 shutdown_connections 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f91277d7640 1 --2- 172.21.15.12:0/22169551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f912006c240 0x7f912006c620 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.894+0000 7f91277d7640 1 -- 172.21.15.12:0/22169551 >> 172.21.15.12:0/22169551 conn(0x7f9120072900 msgr2=0x7f91200d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:17.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.895+0000 7f91277d7640 1 -- 172.21.15.12:0/22169551 shutdown_connections 2024-03-08T08:29:17.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.895+0000 7f91277d7640 1 -- 172.21.15.12:0/22169551 wait complete. 2024-03-08T08:29:17.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 Processor -- start 2024-03-08T08:29:17.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 -- start start 2024-03-08T08:29:17.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 0x7f91200b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91200b9420 0x7f91200b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91200d1f90 con 0x7f91200d24b0 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91200d1cb0 con 0x7f91200b9420 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f91277d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912015bc30 con 0x7f912006c240 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f9125d74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.896+0000 7f9125d74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58188/0 (socket says 172.21.15.12:58188) 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 -- 172.21.15.12:0/1985033482 learned_addr learned my addr 172.21.15.12:0/1985033482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9124d72640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91200b9420 0x7f91200b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125573640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 0x7f91200b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 -- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 msgr2=0x7f91200b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 0x7f91200b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 -- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91200b9420 msgr2=0x7f91200b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:17.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91200b9420 0x7f91200b37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 -- 172.21.15.12:0/1985033482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91200b5bb0 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125573640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 0x7f91200b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9125d74640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f911800c730 tx=0x7f911800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118018070 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911800ce40 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911800a860 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91200cae80 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.897+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91200cb440 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.898+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91200749c0 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.898+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f911801b070 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.899+0000 7f9115ffb640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 0x7f910406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.899+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9118089730 con 0x7f91200d24b0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.899+0000 7f9125573640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 0x7f910406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:17.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.899+0000 7f9125573640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 0x7f910406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f911c036000 tx=0x7f911c039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:17.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:17.903+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f91180585b0 con 0x7f91200d24b0 2024-03-08T08:29:18.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.078+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91200b3210 con 0x7f910406b500 2024-03-08T08:29:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.084+0000 7f9115ffb640 1 -- 172.21.15.12:0/1985033482 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251833 (secure 0 0 0) 0x7f91200b3210 con 0x7f910406b500 2024-03-08T08:29:18.088 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 msgr2=0x7f910406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 0x7f910406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f911c036000 tx=0x7f911c039040 comp rx=0 tx=0).stop 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 msgr2=0x7f91200b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f911800c730 tx=0x7f911800cbf0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 shutdown_connections 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f910406b500 0x7f910406d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200d24b0 0x7f91200b3ce0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91200b9420 0x7f91200b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 --2- 172.21.15.12:0/1985033482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f912006c240 0x7f91200b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 >> 172.21.15.12:0/1985033482 conn(0x7f9120072900 msgr2=0x7f912015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 shutdown_connections 2024-03-08T08:29:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.087+0000 7f91277d7640 1 -- 172.21.15.12:0/1985033482 wait complete. 2024-03-08T08:29:18.099 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-08T08:29:16.995168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239880,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734840,"statfs":{"total":773094113280,"available":772848476160,"internally_reserved":0,"allocated":2957312,"data_stored":1776620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002248"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562754+0000","last_change":"2024-03-08T08:29:06.251346+0000","last_active":"2024-03-08T08:29:07.562754+0000","last_peered":"2024-03-08T08:29:07.562754+0000","last_clean":"2024-03-08T08:29:07.562754+0000","last_became_active":"2024-03-08T08:29:06.250986+0000","last_became_peered":"2024-03-08T08:29:06.250986+0000","last_unstale":"2024-03-08T08:29:07.562754+0000","last_undegraded":"2024-03-08T08:29:07.562754+0000","last_fullsized":"2024-03-08T08:29:07.562754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:31:13.095757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905464+0000","last_change":"2024-03-08T08:29:06.247932+0000","last_active":"2024-03-08T08:29:07.905464+0000","last_peered":"2024-03-08T08:29:07.905464+0000","last_clean":"2024-03-08T08:29:07.905464+0000","last_became_active":"2024-03-08T08:29:06.247623+0000","last_became_peered":"2024-03-08T08:29:06.247623+0000","last_unstale":"2024-03-08T08:29:07.905464+0000","last_undegraded":"2024-03-08T08:29:07.905464+0000","last_fullsized":"2024-03-08T08:29:07.905464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:48:05.979625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905335+0000","last_change":"2024-03-08T08:29:06.244061+0000","last_active":"2024-03-08T08:29:07.905335+0000","last_peered":"2024-03-08T08:29:07.905335+0000","last_clean":"2024-03-08T08:29:07.905335+0000","last_became_active":"2024-03-08T08:29:06.243801+0000","last_became_peered":"2024-03-08T08:29:06.243801+0000","last_unstale":"2024-03-08T08:29:07.905335+0000","last_undegraded":"2024-03-08T08:29:07.905335+0000","last_fullsized":"2024-03-08T08:29:07.905335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:49:40.562208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995367+0000","last_change":"2024-03-08T08:29:06.250895+0000","last_active":"2024-03-08T08:29:09.995367+0000","last_peered":"2024-03-08T08:29:09.995367+0000","last_clean":"2024-03-08T08:29:09.995367+0000","last_became_active":"2024-03-08T08:29:06.250560+0000","last_became_peered":"2024-03-08T08:29:06.250560+0000","last_unstale":"2024-03-08T08:29:09.995367+0000","last_undegraded":"2024-03-08T08:29:09.995367+0000","last_fullsized":"2024-03-08T08:29:09.995367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:22:34.858690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562498+0000","last_change":"2024-03-08T08:29:06.249635+0000","last_active":"2024-03-08T08:29:07.562498+0000","last_peered":"2024-03-08T08:29:07.562498+0000","last_clean":"2024-03-08T08:29:07.562498+0000","last_became_active":"2024-03-08T08:29:06.249356+0000","last_became_peered":"2024-03-08T08:29:06.249356+0000","last_unstale":"2024-03-08T08:29:07.562498+0000","last_undegraded":"2024-03-08T08:29:07.562498+0000","last_fullsized":"2024-03-08T08:29:07.562498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:47:43.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994809+0000","last_change":"2024-03-08T08:29:06.249994+0000","last_active":"2024-03-08T08:29:09.994809+0000","last_peered":"2024-03-08T08:29:09.994809+0000","last_clean":"2024-03-08T08:29:09.994809+0000","last_became_active":"2024-03-08T08:29:06.249682+0000","last_became_peered":"2024-03-08T08:29:06.249682+0000","last_unstale":"2024-03-08T08:29:09.994809+0000","last_undegraded":"2024-03-08T08:29:09.994809+0000","last_fullsized":"2024-03-08T08:29:09.994809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:36.028927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562203+0000","last_change":"2024-03-08T08:29:06.237963+0000","last_active":"2024-03-08T08:29:07.562203+0000","last_peered":"2024-03-08T08:29:07.562203+0000","last_clean":"2024-03-08T08:29:07.562203+0000","last_became_active":"2024-03-08T08:29:06.237654+0000","last_became_peered":"2024-03-08T08:29:06.237654+0000","last_unstale":"2024-03-08T08:29:07.562203+0000","last_undegraded":"2024-03-08T08:29:07.562203+0000","last_fullsized":"2024-03-08T08:29:07.562203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:16:24.171714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905138+0000","last_change":"2024-03-08T08:29:06.258895+0000","last_active":"2024-03-08T08:29:07.905138+0000","last_peered":"2024-03-08T08:29:07.905138+0000","last_clean":"2024-03-08T08:29:07.905138+0000","last_became_active":"2024-03-08T08:29:06.258630+0000","last_became_peered":"2024-03-08T08:29:06.258630+0000","last_unstale":"2024-03-08T08:29:07.905138+0000","last_undegraded":"2024-03-08T08:29:07.905138+0000","last_fullsized":"2024-03-08T08:29:07.905138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:02:10.390227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236928+0000","last_change":"2024-03-08T08:29:06.260239+0000","last_active":"2024-03-08T08:29:07.236928+0000","last_peered":"2024-03-08T08:29:07.236928+0000","last_clean":"2024-03-08T08:29:07.236928+0000","last_became_active":"2024-03-08T08:29:06.259944+0000","last_became_peered":"2024-03-08T08:29:06.259944+0000","last_unstale":"2024-03-08T08:29:07.236928+0000","last_undegraded":"2024-03-08T08:29:07.236928+0000","last_fullsized":"2024-03-08T08:29:07.236928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:18:09.551413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562557+0000","last_change":"2024-03-08T08:29:06.252950+0000","last_active":"2024-03-08T08:29:07.562557+0000","last_peered":"2024-03-08T08:29:07.562557+0000","last_clean":"2024-03-08T08:29:07.562557+0000","last_became_active":"2024-03-08T08:29:06.252480+0000","last_became_peered":"2024-03-08T08:29:06.252480+0000","last_unstale":"2024-03-08T08:29:07.562557+0000","last_undegraded":"2024-03-08T08:29:07.562557+0000","last_fullsized":"2024-03-08T08:29:07.562557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:40:11.278068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562751+0000","last_change":"2024-03-08T08:29:06.248718+0000","last_active":"2024-03-08T08:29:07.562751+0000","last_peered":"2024-03-08T08:29:07.562751+0000","last_clean":"2024-03-08T08:29:07.562751+0000","last_became_active":"2024-03-08T08:29:06.248400+0000","last_became_peered":"2024-03-08T08:29:06.248400+0000","last_unstale":"2024-03-08T08:29:07.562751+0000","last_undegraded":"2024-03-08T08:29:07.562751+0000","last_fullsized":"2024-03-08T08:29:07.562751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:02:07.589132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562439+0000","last_change":"2024-03-08T08:29:06.247528+0000","last_active":"2024-03-08T08:29:07.562439+0000","last_peered":"2024-03-08T08:29:07.562439+0000","last_clean":"2024-03-08T08:29:07.562439+0000","last_became_active":"2024-03-08T08:29:06.246926+0000","last_became_peered":"2024-03-08T08:29:06.246926+0000","last_unstale":"2024-03-08T08:29:07.562439+0000","last_undegraded":"2024-03-08T08:29:07.562439+0000","last_fullsized":"2024-03-08T08:29:07.562439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:42.233464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409364+0000","last_change":"2024-03-08T08:29:06.261811+0000","last_active":"2024-03-08T08:29:07.409364+0000","last_peered":"2024-03-08T08:29:07.409364+0000","last_clean":"2024-03-08T08:29:07.409364+0000","last_became_active":"2024-03-08T08:29:06.261568+0000","last_became_peered":"2024-03-08T08:29:06.261568+0000","last_unstale":"2024-03-08T08:29:07.409364+0000","last_undegraded":"2024-03-08T08:29:07.409364+0000","last_fullsized":"2024-03-08T08:29:07.409364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409154+0000","last_change":"2024-03-08T08:29:06.245702+0000","last_active":"2024-03-08T08:29:07.409154+0000","last_peered":"2024-03-08T08:29:07.409154+0000","last_clean":"2024-03-08T08:29:07.409154+0000","last_became_active":"2024-03-08T08:29:06.245425+0000","last_became_peered":"2024-03-08T08:29:06.245425+0000","last_unstale":"2024-03-08T08:29:07.409154+0000","last_undegraded":"2024-03-08T08:29:07.409154+0000","last_fullsized":"2024-03-08T08:29:07.409154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:01.699386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905113+0000","last_change":"2024-03-08T08:29:06.250908+0000","last_active":"2024-03-08T08:29:07.905113+0000","last_peered":"2024-03-08T08:29:07.905113+0000","last_clean":"2024-03-08T08:29:07.905113+0000","last_became_active":"2024-03-08T08:29:06.250627+0000","last_became_peered":"2024-03-08T08:29:06.250627+0000","last_unstale":"2024-03-08T08:29:07.905113+0000","last_undegraded":"2024-03-08T08:29:07.905113+0000","last_fullsized":"2024-03-08T08:29:07.905113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:26:00.214895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994982+0000","last_change":"2024-03-08T08:29:06.257106+0000","last_active":"2024-03-08T08:29:09.994982+0000","last_peered":"2024-03-08T08:29:09.994982+0000","last_clean":"2024-03-08T08:29:09.994982+0000","last_became_active":"2024-03-08T08:29:06.256817+0000","last_became_peered":"2024-03-08T08:29:06.256817+0000","last_unstale":"2024-03-08T08:29:09.994982+0000","last_undegraded":"2024-03-08T08:29:09.994982+0000","last_fullsized":"2024-03-08T08:29:09.994982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:02:07.589132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995264+0000","last_change":"2024-03-08T08:29:06.262604+0000","last_active":"2024-03-08T08:29:09.995264+0000","last_peered":"2024-03-08T08:29:09.995264+0000","last_clean":"2024-03-08T08:29:09.995264+0000","last_became_active":"2024-03-08T08:29:06.262222+0000","last_became_peered":"2024-03-08T08:29:06.262222+0000","last_unstale":"2024-03-08T08:29:09.995264+0000","last_undegraded":"2024-03-08T08:29:09.995264+0000","last_fullsized":"2024-03-08T08:29:09.995264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:05:14.678904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235999+0000","last_change":"2024-03-08T08:29:06.256729+0000","last_active":"2024-03-08T08:29:07.235999+0000","last_peered":"2024-03-08T08:29:07.235999+0000","last_clean":"2024-03-08T08:29:07.235999+0000","last_became_active":"2024-03-08T08:29:06.256220+0000","last_became_peered":"2024-03-08T08:29:06.256220+0000","last_unstale":"2024-03-08T08:29:07.235999+0000","last_undegraded":"2024-03-08T08:29:07.235999+0000","last_fullsized":"2024-03-08T08:29:07.235999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:16.445063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995649+0000","last_change":"2024-03-08T08:29:06.251913+0000","last_active":"2024-03-08T08:29:09.995649+0000","last_peered":"2024-03-08T08:29:09.995649+0000","last_clean":"2024-03-08T08:29:09.995649+0000","last_became_active":"2024-03-08T08:29:06.251609+0000","last_became_peered":"2024-03-08T08:29:06.251609+0000","last_unstale":"2024-03-08T08:29:09.995649+0000","last_undegraded":"2024-03-08T08:29:09.995649+0000","last_fullsized":"2024-03-08T08:29:09.995649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:17:00.254942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562581+0000","last_change":"2024-03-08T08:29:06.247738+0000","last_active":"2024-03-08T08:29:07.562581+0000","last_peered":"2024-03-08T08:29:07.562581+0000","last_clean":"2024-03-08T08:29:07.562581+0000","last_became_active":"2024-03-08T08:29:06.247254+0000","last_became_peered":"2024-03-08T08:29:06.247254+0000","last_unstale":"2024-03-08T08:29:07.562581+0000","last_undegraded":"2024-03-08T08:29:07.562581+0000","last_fullsized":"2024-03-08T08:29:07.562581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:01:26.719267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235275+0000","last_change":"2024-03-08T08:29:06.251015+0000","last_active":"2024-03-08T08:29:07.235275+0000","last_peered":"2024-03-08T08:29:07.235275+0000","last_clean":"2024-03-08T08:29:07.235275+0000","last_became_active":"2024-03-08T08:29:06.250716+0000","last_became_peered":"2024-03-08T08:29:06.250716+0000","last_unstale":"2024-03-08T08:29:07.235275+0000","last_undegraded":"2024-03-08T08:29:07.235275+0000","last_fullsized":"2024-03-08T08:29:07.235275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236632+0000","last_change":"2024-03-08T08:29:06.259314+0000","last_active":"2024-03-08T08:29:07.236632+0000","last_peered":"2024-03-08T08:29:07.236632+0000","last_clean":"2024-03-08T08:29:07.236632+0000","last_became_active":"2024-03-08T08:29:06.258967+0000","last_became_peered":"2024-03-08T08:29:06.258967+0000","last_unstale":"2024-03-08T08:29:07.236632+0000","last_undegraded":"2024-03-08T08:29:07.236632+0000","last_fullsized":"2024-03-08T08:29:07.236632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:04:34.787442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905406+0000","last_change":"2024-03-08T08:29:06.251496+0000","last_active":"2024-03-08T08:29:07.905406+0000","last_peered":"2024-03-08T08:29:07.905406+0000","last_clean":"2024-03-08T08:29:07.905406+0000","last_became_active":"2024-03-08T08:29:06.251220+0000","last_became_peered":"2024-03-08T08:29:06.251220+0000","last_unstale":"2024-03-08T08:29:07.905406+0000","last_undegraded":"2024-03-08T08:29:07.905406+0000","last_fullsized":"2024-03-08T08:29:07.905406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:31:41.063261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994650+0000","last_change":"2024-03-08T08:29:06.256404+0000","last_active":"2024-03-08T08:29:09.994650+0000","last_peered":"2024-03-08T08:29:09.994650+0000","last_clean":"2024-03-08T08:29:09.994650+0000","last_became_active":"2024-03-08T08:29:06.256087+0000","last_became_peered":"2024-03-08T08:29:06.256087+0000","last_unstale":"2024-03-08T08:29:09.994650+0000","last_undegraded":"2024-03-08T08:29:09.994650+0000","last_fullsized":"2024-03-08T08:29:09.994650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:01:26.719267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905562+0000","last_change":"2024-03-08T08:29:06.255787+0000","last_active":"2024-03-08T08:29:07.905562+0000","last_peered":"2024-03-08T08:29:07.905562+0000","last_clean":"2024-03-08T08:29:07.905562+0000","last_became_active":"2024-03-08T08:29:06.255548+0000","last_became_peered":"2024-03-08T08:29:06.255548+0000","last_unstale":"2024-03-08T08:29:07.905562+0000","last_undegraded":"2024-03-08T08:29:07.905562+0000","last_fullsized":"2024-03-08T08:29:07.905562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:28:26.547413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995638+0000","last_change":"2024-03-08T08:29:06.259612+0000","last_active":"2024-03-08T08:29:09.995638+0000","last_peered":"2024-03-08T08:29:09.995638+0000","last_clean":"2024-03-08T08:29:09.995638+0000","last_became_active":"2024-03-08T08:29:06.259198+0000","last_became_peered":"2024-03-08T08:29:06.259198+0000","last_unstale":"2024-03-08T08:29:09.995638+0000","last_undegraded":"2024-03-08T08:29:09.995638+0000","last_fullsized":"2024-03-08T08:29:09.995638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:05.929603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409477+0000","last_change":"2024-03-08T08:29:06.245080+0000","last_active":"2024-03-08T08:29:07.409477+0000","last_peered":"2024-03-08T08:29:07.409477+0000","last_clean":"2024-03-08T08:29:07.409477+0000","last_became_active":"2024-03-08T08:29:06.244821+0000","last_became_peered":"2024-03-08T08:29:06.244821+0000","last_unstale":"2024-03-08T08:29:07.409477+0000","last_undegraded":"2024-03-08T08:29:07.409477+0000","last_fullsized":"2024-03-08T08:29:07.409477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409429+0000","last_change":"2024-03-08T08:29:06.259724+0000","last_active":"2024-03-08T08:29:07.409429+0000","last_peered":"2024-03-08T08:29:07.409429+0000","last_clean":"2024-03-08T08:29:07.409429+0000","last_became_active":"2024-03-08T08:29:06.259415+0000","last_became_peered":"2024-03-08T08:29:06.259415+0000","last_unstale":"2024-03-08T08:29:07.409429+0000","last_undegraded":"2024-03-08T08:29:07.409429+0000","last_fullsized":"2024-03-08T08:29:07.409429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905249+0000","last_change":"2024-03-08T08:29:06.248138+0000","last_active":"2024-03-08T08:29:07.905249+0000","last_peered":"2024-03-08T08:29:07.905249+0000","last_clean":"2024-03-08T08:29:07.905249+0000","last_became_active":"2024-03-08T08:29:06.247330+0000","last_became_peered":"2024-03-08T08:29:06.247330+0000","last_unstale":"2024-03-08T08:29:07.905249+0000","last_undegraded":"2024-03-08T08:29:07.905249+0000","last_fullsized":"2024-03-08T08:29:07.905249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:40:15.729074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995284+0000","last_change":"2024-03-08T08:29:06.254998+0000","last_active":"2024-03-08T08:29:09.995284+0000","last_peered":"2024-03-08T08:29:09.995284+0000","last_clean":"2024-03-08T08:29:09.995284+0000","last_became_active":"2024-03-08T08:29:06.254596+0000","last_became_peered":"2024-03-08T08:29:06.254596+0000","last_unstale":"2024-03-08T08:29:09.995284+0000","last_undegraded":"2024-03-08T08:29:09.995284+0000","last_fullsized":"2024-03-08T08:29:09.995284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:04.577609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235098+0000","last_change":"2024-03-08T08:29:06.244995+0000","last_active":"2024-03-08T08:29:07.235098+0000","last_peered":"2024-03-08T08:29:07.235098+0000","last_clean":"2024-03-08T08:29:07.235098+0000","last_became_active":"2024-03-08T08:29:06.244712+0000","last_became_peered":"2024-03-08T08:29:06.244712+0000","last_unstale":"2024-03-08T08:29:07.235098+0000","last_undegraded":"2024-03-08T08:29:07.235098+0000","last_fullsized":"2024-03-08T08:29:07.235098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:23.178032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994651+0000","last_change":"2024-03-08T08:29:06.255239+0000","last_active":"2024-03-08T08:29:09.994651+0000","last_peered":"2024-03-08T08:29:09.994651+0000","last_clean":"2024-03-08T08:29:09.994651+0000","last_became_active":"2024-03-08T08:29:06.254635+0000","last_became_peered":"2024-03-08T08:29:06.254635+0000","last_unstale":"2024-03-08T08:29:09.994651+0000","last_undegraded":"2024-03-08T08:29:09.994651+0000","last_fullsized":"2024-03-08T08:29:09.994651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:23.178032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236746+0000","last_change":"2024-03-08T08:29:06.252849+0000","last_active":"2024-03-08T08:29:07.236746+0000","last_peered":"2024-03-08T08:29:07.236746+0000","last_clean":"2024-03-08T08:29:07.236746+0000","last_became_active":"2024-03-08T08:29:06.252559+0000","last_became_peered":"2024-03-08T08:29:06.252559+0000","last_unstale":"2024-03-08T08:29:07.236746+0000","last_undegraded":"2024-03-08T08:29:07.236746+0000","last_fullsized":"2024-03-08T08:29:07.236746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:06:45.449613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905459+0000","last_change":"2024-03-08T08:29:05.247749+0000","last_active":"2024-03-08T08:29:07.905459+0000","last_peered":"2024-03-08T08:29:07.905459+0000","last_clean":"2024-03-08T08:29:07.905459+0000","last_became_active":"2024-03-08T08:29:05.247492+0000","last_became_peered":"2024-03-08T08:29:05.247492+0000","last_unstale":"2024-03-08T08:29:07.905459+0000","last_undegraded":"2024-03-08T08:29:07.905459+0000","last_fullsized":"2024-03-08T08:29:07.905459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:14:48.392940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235447+0000","last_change":"2024-03-08T08:29:06.248631+0000","last_active":"2024-03-08T08:29:07.235447+0000","last_peered":"2024-03-08T08:29:07.235447+0000","last_clean":"2024-03-08T08:29:07.235447+0000","last_became_active":"2024-03-08T08:29:06.248337+0000","last_became_peered":"2024-03-08T08:29:06.248337+0000","last_unstale":"2024-03-08T08:29:07.235447+0000","last_undegraded":"2024-03-08T08:29:07.235447+0000","last_fullsized":"2024-03-08T08:29:07.235447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995383+0000","last_change":"2024-03-08T08:29:05.256580+0000","last_active":"2024-03-08T08:29:09.995383+0000","last_peered":"2024-03-08T08:29:09.995383+0000","last_clean":"2024-03-08T08:29:09.995383+0000","last_became_active":"2024-03-08T08:29:05.256286+0000","last_became_peered":"2024-03-08T08:29:05.256286+0000","last_unstale":"2024-03-08T08:29:09.995383+0000","last_undegraded":"2024-03-08T08:29:09.995383+0000","last_fullsized":"2024-03-08T08:29:09.995383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409076+0000","last_change":"2024-03-08T08:29:05.253153+0000","last_active":"2024-03-08T08:29:07.409076+0000","last_peered":"2024-03-08T08:29:07.409076+0000","last_clean":"2024-03-08T08:29:07.409076+0000","last_became_active":"2024-03-08T08:29:05.252819+0000","last_became_peered":"2024-03-08T08:29:05.252819+0000","last_unstale":"2024-03-08T08:29:07.409076+0000","last_undegraded":"2024-03-08T08:29:07.409076+0000","last_fullsized":"2024-03-08T08:29:07.409076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905491+0000","last_change":"2024-03-08T08:29:06.249692+0000","last_active":"2024-03-08T08:29:07.905491+0000","last_peered":"2024-03-08T08:29:07.905491+0000","last_clean":"2024-03-08T08:29:07.905491+0000","last_became_active":"2024-03-08T08:29:06.248953+0000","last_became_peered":"2024-03-08T08:29:06.248953+0000","last_unstale":"2024-03-08T08:29:07.905491+0000","last_undegraded":"2024-03-08T08:29:07.905491+0000","last_fullsized":"2024-03-08T08:29:07.905491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:30:01.675268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994744+0000","last_change":"2024-03-08T08:29:05.257702+0000","last_active":"2024-03-08T08:29:09.994744+0000","last_peered":"2024-03-08T08:29:09.994744+0000","last_clean":"2024-03-08T08:29:09.994744+0000","last_became_active":"2024-03-08T08:29:05.257406+0000","last_became_peered":"2024-03-08T08:29:05.257406+0000","last_unstale":"2024-03-08T08:29:09.994744+0000","last_undegraded":"2024-03-08T08:29:09.994744+0000","last_fullsized":"2024-03-08T08:29:09.994744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:43.430990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235273+0000","last_change":"2024-03-08T08:29:06.246139+0000","last_active":"2024-03-08T08:29:07.235273+0000","last_peered":"2024-03-08T08:29:07.235273+0000","last_clean":"2024-03-08T08:29:07.235273+0000","last_became_active":"2024-03-08T08:29:06.245660+0000","last_became_peered":"2024-03-08T08:29:06.245660+0000","last_unstale":"2024-03-08T08:29:07.235273+0000","last_undegraded":"2024-03-08T08:29:07.235273+0000","last_fullsized":"2024-03-08T08:29:07.235273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236862+0000","last_change":"2024-03-08T08:29:05.240557+0000","last_active":"2024-03-08T08:29:07.236862+0000","last_peered":"2024-03-08T08:29:07.236862+0000","last_clean":"2024-03-08T08:29:07.236862+0000","last_became_active":"2024-03-08T08:29:05.240233+0000","last_became_peered":"2024-03-08T08:29:05.240233+0000","last_unstale":"2024-03-08T08:29:07.236862+0000","last_undegraded":"2024-03-08T08:29:07.236862+0000","last_fullsized":"2024-03-08T08:29:07.236862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995697+0000","last_change":"2024-03-08T08:29:06.258421+0000","last_active":"2024-03-08T08:29:09.995697+0000","last_peered":"2024-03-08T08:29:09.995697+0000","last_clean":"2024-03-08T08:29:09.995697+0000","last_became_active":"2024-03-08T08:29:06.258079+0000","last_became_peered":"2024-03-08T08:29:06.258079+0000","last_unstale":"2024-03-08T08:29:09.995697+0000","last_undegraded":"2024-03-08T08:29:09.995697+0000","last_fullsized":"2024-03-08T08:29:09.995697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:18:31.433508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562645+0000","last_change":"2024-03-08T08:29:05.251488+0000","last_active":"2024-03-08T08:29:07.562645+0000","last_peered":"2024-03-08T08:29:07.562645+0000","last_clean":"2024-03-08T08:29:07.562645+0000","last_became_active":"2024-03-08T08:29:05.251220+0000","last_became_peered":"2024-03-08T08:29:05.251220+0000","last_unstale":"2024-03-08T08:29:07.562645+0000","last_undegraded":"2024-03-08T08:29:07.562645+0000","last_fullsized":"2024-03-08T08:29:07.562645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905718+0000","last_change":"2024-03-08T08:29:06.257799+0000","last_active":"2024-03-08T08:29:07.905718+0000","last_peered":"2024-03-08T08:29:07.905718+0000","last_clean":"2024-03-08T08:29:07.905718+0000","last_became_active":"2024-03-08T08:29:06.257545+0000","last_became_peered":"2024-03-08T08:29:06.257545+0000","last_unstale":"2024-03-08T08:29:07.905718+0000","last_undegraded":"2024-03-08T08:29:07.905718+0000","last_fullsized":"2024-03-08T08:29:07.905718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:21:44.744902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236028+0000","last_change":"2024-03-08T08:29:05.240046+0000","last_active":"2024-03-08T08:29:07.236028+0000","last_peered":"2024-03-08T08:29:07.236028+0000","last_clean":"2024-03-08T08:29:07.236028+0000","last_became_active":"2024-03-08T08:29:05.239754+0000","last_became_peered":"2024-03-08T08:29:05.239754+0000","last_unstale":"2024-03-08T08:29:07.236028+0000","last_undegraded":"2024-03-08T08:29:07.236028+0000","last_fullsized":"2024-03-08T08:29:07.236028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994857+0000","last_change":"2024-03-08T08:29:06.258228+0000","last_active":"2024-03-08T08:29:09.994857+0000","last_peered":"2024-03-08T08:29:09.994857+0000","last_clean":"2024-03-08T08:29:09.994857+0000","last_became_active":"2024-03-08T08:29:06.256849+0000","last_became_peered":"2024-03-08T08:29:06.256849+0000","last_unstale":"2024-03-08T08:29:09.994857+0000","last_undegraded":"2024-03-08T08:29:09.994857+0000","last_fullsized":"2024-03-08T08:29:09.994857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:31:13.095757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995463+0000","last_change":"2024-03-08T08:29:05.246882+0000","last_active":"2024-03-08T08:29:09.995463+0000","last_peered":"2024-03-08T08:29:09.995463+0000","last_clean":"2024-03-08T08:29:09.995463+0000","last_became_active":"2024-03-08T08:29:05.240261+0000","last_became_peered":"2024-03-08T08:29:05.240261+0000","last_unstale":"2024-03-08T08:29:09.995463+0000","last_undegraded":"2024-03-08T08:29:09.995463+0000","last_fullsized":"2024-03-08T08:29:09.995463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:20:06.037325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562506+0000","last_change":"2024-03-08T08:29:06.246529+0000","last_active":"2024-03-08T08:29:07.562506+0000","last_peered":"2024-03-08T08:29:07.562506+0000","last_clean":"2024-03-08T08:29:07.562506+0000","last_became_active":"2024-03-08T08:29:06.246230+0000","last_became_peered":"2024-03-08T08:29:06.246230+0000","last_unstale":"2024-03-08T08:29:07.562506+0000","last_undegraded":"2024-03-08T08:29:07.562506+0000","last_fullsized":"2024-03-08T08:29:07.562506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:23:31.113352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235779+0000","last_change":"2024-03-08T08:29:05.251723+0000","last_active":"2024-03-08T08:29:07.235779+0000","last_peered":"2024-03-08T08:29:07.235779+0000","last_clean":"2024-03-08T08:29:07.235779+0000","last_became_active":"2024-03-08T08:29:05.251410+0000","last_became_peered":"2024-03-08T08:29:05.251410+0000","last_unstale":"2024-03-08T08:29:07.235779+0000","last_undegraded":"2024-03-08T08:29:07.235779+0000","last_fullsized":"2024-03-08T08:29:07.235779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409264+0000","last_change":"2024-03-08T08:29:06.249661+0000","last_active":"2024-03-08T08:29:07.409264+0000","last_peered":"2024-03-08T08:29:07.409264+0000","last_clean":"2024-03-08T08:29:07.409264+0000","last_became_active":"2024-03-08T08:29:06.249388+0000","last_became_peered":"2024-03-08T08:29:06.249388+0000","last_unstale":"2024-03-08T08:29:07.409264+0000","last_undegraded":"2024-03-08T08:29:07.409264+0000","last_fullsized":"2024-03-08T08:29:07.409264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995432+0000","last_change":"2024-03-08T08:29:05.250954+0000","last_active":"2024-03-08T08:29:09.995432+0000","last_peered":"2024-03-08T08:29:09.995432+0000","last_clean":"2024-03-08T08:29:09.995432+0000","last_became_active":"2024-03-08T08:29:05.250478+0000","last_became_peered":"2024-03-08T08:29:05.250478+0000","last_unstale":"2024-03-08T08:29:09.995432+0000","last_undegraded":"2024-03-08T08:29:09.995432+0000","last_fullsized":"2024-03-08T08:29:09.995432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:22:33.853562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994832+0000","last_change":"2024-03-08T08:29:06.263756+0000","last_active":"2024-03-08T08:29:09.994832+0000","last_peered":"2024-03-08T08:29:09.994832+0000","last_clean":"2024-03-08T08:29:09.994832+0000","last_became_active":"2024-03-08T08:29:06.263461+0000","last_became_peered":"2024-03-08T08:29:06.263461+0000","last_unstale":"2024-03-08T08:29:09.994832+0000","last_undegraded":"2024-03-08T08:29:09.994832+0000","last_fullsized":"2024-03-08T08:29:09.994832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:37:49.534588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562192+0000","last_change":"2024-03-08T08:29:05.253842+0000","last_active":"2024-03-08T08:29:07.562192+0000","last_peered":"2024-03-08T08:29:07.562192+0000","last_clean":"2024-03-08T08:29:07.562192+0000","last_became_active":"2024-03-08T08:29:05.253552+0000","last_became_peered":"2024-03-08T08:29:05.253552+0000","last_unstale":"2024-03-08T08:29:07.562192+0000","last_undegraded":"2024-03-08T08:29:07.562192+0000","last_fullsized":"2024-03-08T08:29:07.562192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236519+0000","last_change":"2024-03-08T08:29:06.259240+0000","last_active":"2024-03-08T08:29:07.236519+0000","last_peered":"2024-03-08T08:29:07.236519+0000","last_clean":"2024-03-08T08:29:07.236519+0000","last_became_active":"2024-03-08T08:29:06.257182+0000","last_became_peered":"2024-03-08T08:29:06.257182+0000","last_unstale":"2024-03-08T08:29:07.236519+0000","last_undegraded":"2024-03-08T08:29:07.236519+0000","last_fullsized":"2024-03-08T08:29:07.236519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:58:34.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235055+0000","last_change":"2024-03-08T08:29:05.237318+0000","last_active":"2024-03-08T08:29:07.235055+0000","last_peered":"2024-03-08T08:29:07.235055+0000","last_clean":"2024-03-08T08:29:07.235055+0000","last_became_active":"2024-03-08T08:29:05.236985+0000","last_became_peered":"2024-03-08T08:29:05.236985+0000","last_unstale":"2024-03-08T08:29:07.235055+0000","last_undegraded":"2024-03-08T08:29:07.235055+0000","last_fullsized":"2024-03-08T08:29:07.235055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995565+0000","last_change":"2024-03-08T08:29:06.256312+0000","last_active":"2024-03-08T08:29:09.995565+0000","last_peered":"2024-03-08T08:29:09.995565+0000","last_clean":"2024-03-08T08:29:09.995565+0000","last_became_active":"2024-03-08T08:29:06.256021+0000","last_became_peered":"2024-03-08T08:29:06.256021+0000","last_unstale":"2024-03-08T08:29:09.995565+0000","last_undegraded":"2024-03-08T08:29:09.995565+0000","last_fullsized":"2024-03-08T08:29:09.995565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:22.530837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562837+0000","last_change":"2024-03-08T08:29:05.253285+0000","last_active":"2024-03-08T08:29:07.562837+0000","last_peered":"2024-03-08T08:29:07.562837+0000","last_clean":"2024-03-08T08:29:07.562837+0000","last_became_active":"2024-03-08T08:29:05.253023+0000","last_became_peered":"2024-03-08T08:29:05.253023+0000","last_unstale":"2024-03-08T08:29:07.562837+0000","last_undegraded":"2024-03-08T08:29:07.562837+0000","last_fullsized":"2024-03-08T08:29:07.562837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995783+0000","last_change":"2024-03-08T08:29:06.263316+0000","last_active":"2024-03-08T08:29:09.995783+0000","last_peered":"2024-03-08T08:29:09.995783+0000","last_clean":"2024-03-08T08:29:09.995783+0000","last_became_active":"2024-03-08T08:29:06.263033+0000","last_became_peered":"2024-03-08T08:29:06.263033+0000","last_unstale":"2024-03-08T08:29:09.995783+0000","last_undegraded":"2024-03-08T08:29:09.995783+0000","last_fullsized":"2024-03-08T08:29:09.995783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:01.699386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.247430+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.247146+0000","last_became_peered":"2024-03-08T08:29:05.247146+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562783+0000","last_change":"2024-03-08T08:29:06.246687+0000","last_active":"2024-03-08T08:29:07.562783+0000","last_peered":"2024-03-08T08:29:07.562783+0000","last_clean":"2024-03-08T08:29:07.562783+0000","last_became_active":"2024-03-08T08:29:06.246421+0000","last_became_peered":"2024-03-08T08:29:06.246421+0000","last_unstale":"2024-03-08T08:29:07.562783+0000","last_undegraded":"2024-03-08T08:29:07.562783+0000","last_fullsized":"2024-03-08T08:29:07.562783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:47:29.808023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905309+0000","last_change":"2024-03-08T08:29:05.246842+0000","last_active":"2024-03-08T08:29:07.905309+0000","last_peered":"2024-03-08T08:29:07.905309+0000","last_clean":"2024-03-08T08:29:07.905309+0000","last_became_active":"2024-03-08T08:29:05.246589+0000","last_became_peered":"2024-03-08T08:29:05.246589+0000","last_unstale":"2024-03-08T08:29:07.905309+0000","last_undegraded":"2024-03-08T08:29:07.905309+0000","last_fullsized":"2024-03-08T08:29:07.905309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:34:13.125245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562749+0000","last_change":"2024-03-08T08:29:06.250869+0000","last_active":"2024-03-08T08:29:07.562749+0000","last_peered":"2024-03-08T08:29:07.562749+0000","last_clean":"2024-03-08T08:29:07.562749+0000","last_became_active":"2024-03-08T08:29:06.250598+0000","last_became_peered":"2024-03-08T08:29:06.250598+0000","last_unstale":"2024-03-08T08:29:07.562749+0000","last_undegraded":"2024-03-08T08:29:07.562749+0000","last_fullsized":"2024-03-08T08:29:07.562749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:37:49.534588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409554+0000","last_change":"2024-03-08T08:29:05.253164+0000","last_active":"2024-03-08T08:29:07.409554+0000","last_peered":"2024-03-08T08:29:07.409554+0000","last_clean":"2024-03-08T08:29:07.409554+0000","last_became_active":"2024-03-08T08:29:05.252833+0000","last_became_peered":"2024-03-08T08:29:05.252833+0000","last_unstale":"2024-03-08T08:29:07.409554+0000","last_undegraded":"2024-03-08T08:29:07.409554+0000","last_fullsized":"2024-03-08T08:29:07.409554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995324+0000","last_change":"2024-03-08T08:29:06.259015+0000","last_active":"2024-03-08T08:29:09.995324+0000","last_peered":"2024-03-08T08:29:09.995324+0000","last_clean":"2024-03-08T08:29:09.995324+0000","last_became_active":"2024-03-08T08:29:06.258632+0000","last_became_peered":"2024-03-08T08:29:06.258632+0000","last_unstale":"2024-03-08T08:29:09.995324+0000","last_undegraded":"2024-03-08T08:29:09.995324+0000","last_fullsized":"2024-03-08T08:29:09.995324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:29:38.770610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236450+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:07.236450+0000","last_peered":"2024-03-08T08:29:07.236450+0000","last_clean":"2024-03-08T08:29:07.236450+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:07.236450+0000","last_undegraded":"2024-03-08T08:29:07.236450+0000","last_fullsized":"2024-03-08T08:29:07.236450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995046+0000","last_change":"2024-03-08T08:29:05.257931+0000","last_active":"2024-03-08T08:29:09.995046+0000","last_peered":"2024-03-08T08:29:09.995046+0000","last_clean":"2024-03-08T08:29:09.995046+0000","last_became_active":"2024-03-08T08:29:05.257637+0000","last_became_peered":"2024-03-08T08:29:05.257637+0000","last_unstale":"2024-03-08T08:29:09.995046+0000","last_undegraded":"2024-03-08T08:29:09.995046+0000","last_fullsized":"2024-03-08T08:29:09.995046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:00.694258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235934+0000","last_change":"2024-03-08T08:29:06.251166+0000","last_active":"2024-03-08T08:29:07.235934+0000","last_peered":"2024-03-08T08:29:07.235934+0000","last_clean":"2024-03-08T08:29:07.235934+0000","last_became_active":"2024-03-08T08:29:06.250778+0000","last_became_peered":"2024-03-08T08:29:06.250778+0000","last_unstale":"2024-03-08T08:29:07.235934+0000","last_undegraded":"2024-03-08T08:29:07.235934+0000","last_fullsized":"2024-03-08T08:29:07.235934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:14:53.496011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994847+0000","last_change":"2024-03-08T08:29:05.251244+0000","last_active":"2024-03-08T08:29:09.994847+0000","last_peered":"2024-03-08T08:29:09.994847+0000","last_clean":"2024-03-08T08:29:09.994847+0000","last_became_active":"2024-03-08T08:29:05.250928+0000","last_became_peered":"2024-03-08T08:29:05.250928+0000","last_unstale":"2024-03-08T08:29:09.994847+0000","last_undegraded":"2024-03-08T08:29:09.994847+0000","last_fullsized":"2024-03-08T08:29:09.994847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:52.269276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409152+0000","last_change":"2024-03-08T08:29:06.255405+0000","last_active":"2024-03-08T08:29:07.409152+0000","last_peered":"2024-03-08T08:29:07.409152+0000","last_clean":"2024-03-08T08:29:07.409152+0000","last_became_active":"2024-03-08T08:29:06.255118+0000","last_became_peered":"2024-03-08T08:29:06.255118+0000","last_unstale":"2024-03-08T08:29:07.409152+0000","last_undegraded":"2024-03-08T08:29:07.409152+0000","last_fullsized":"2024-03-08T08:29:07.409152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:44.436118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235211+0000","last_change":"2024-03-08T08:29:05.245447+0000","last_active":"2024-03-08T08:29:07.235211+0000","last_peered":"2024-03-08T08:29:07.235211+0000","last_clean":"2024-03-08T08:29:07.235211+0000","last_became_active":"2024-03-08T08:29:05.245148+0000","last_became_peered":"2024-03-08T08:29:05.245148+0000","last_unstale":"2024-03-08T08:29:07.235211+0000","last_undegraded":"2024-03-08T08:29:07.235211+0000","last_fullsized":"2024-03-08T08:29:07.235211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905516+0000","last_change":"2024-03-08T08:29:06.248546+0000","last_active":"2024-03-08T08:29:07.905516+0000","last_peered":"2024-03-08T08:29:07.905516+0000","last_clean":"2024-03-08T08:29:07.905516+0000","last_became_active":"2024-03-08T08:29:06.248292+0000","last_became_peered":"2024-03-08T08:29:06.248292+0000","last_unstale":"2024-03-08T08:29:07.905516+0000","last_undegraded":"2024-03-08T08:29:07.905516+0000","last_fullsized":"2024-03-08T08:29:07.905516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:32:56.591618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995527+0000","last_change":"2024-03-08T08:29:05.250674+0000","last_active":"2024-03-08T08:29:09.995527+0000","last_peered":"2024-03-08T08:29:09.995527+0000","last_clean":"2024-03-08T08:29:09.995527+0000","last_became_active":"2024-03-08T08:29:05.250342+0000","last_became_peered":"2024-03-08T08:29:05.250342+0000","last_unstale":"2024-03-08T08:29:09.995527+0000","last_undegraded":"2024-03-08T08:29:09.995527+0000","last_fullsized":"2024-03-08T08:29:09.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:04.924475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562458+0000","last_change":"2024-03-08T08:29:06.252736+0000","last_active":"2024-03-08T08:29:07.562458+0000","last_peered":"2024-03-08T08:29:07.562458+0000","last_clean":"2024-03-08T08:29:07.562458+0000","last_became_active":"2024-03-08T08:29:06.251566+0000","last_became_peered":"2024-03-08T08:29:06.251566+0000","last_unstale":"2024-03-08T08:29:07.562458+0000","last_undegraded":"2024-03-08T08:29:07.562458+0000","last_fullsized":"2024-03-08T08:29:07.562458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:52:02.676611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905374+0000","last_change":"2024-03-08T08:29:05.245146+0000","last_active":"2024-03-08T08:29:07.905374+0000","last_peered":"2024-03-08T08:29:07.905374+0000","last_clean":"2024-03-08T08:29:07.905374+0000","last_became_active":"2024-03-08T08:29:05.244812+0000","last_became_peered":"2024-03-08T08:29:05.244812+0000","last_unstale":"2024-03-08T08:29:07.905374+0000","last_undegraded":"2024-03-08T08:29:07.905374+0000","last_fullsized":"2024-03-08T08:29:07.905374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994818+0000","last_change":"2024-03-08T08:29:06.254952+0000","last_active":"2024-03-08T08:29:09.994818+0000","last_peered":"2024-03-08T08:29:09.994818+0000","last_clean":"2024-03-08T08:29:09.994818+0000","last_became_active":"2024-03-08T08:29:06.254408+0000","last_became_peered":"2024-03-08T08:29:06.254408+0000","last_unstale":"2024-03-08T08:29:09.994818+0000","last_undegraded":"2024-03-08T08:29:09.994818+0000","last_fullsized":"2024-03-08T08:29:09.994818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:16.445063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995531+0000","last_change":"2024-03-08T08:29:05.256587+0000","last_active":"2024-03-08T08:29:09.995531+0000","last_peered":"2024-03-08T08:29:09.995531+0000","last_clean":"2024-03-08T08:29:09.995531+0000","last_became_active":"2024-03-08T08:29:05.256091+0000","last_became_peered":"2024-03-08T08:29:05.256091+0000","last_unstale":"2024-03-08T08:29:09.995531+0000","last_undegraded":"2024-03-08T08:29:09.995531+0000","last_fullsized":"2024-03-08T08:29:09.995531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409456+0000","last_change":"2024-03-08T08:29:06.250994+0000","last_active":"2024-03-08T08:29:07.409456+0000","last_peered":"2024-03-08T08:29:07.409456+0000","last_clean":"2024-03-08T08:29:07.409456+0000","last_became_active":"2024-03-08T08:29:06.250738+0000","last_became_peered":"2024-03-08T08:29:06.250738+0000","last_unstale":"2024-03-08T08:29:07.409456+0000","last_undegraded":"2024-03-08T08:29:07.409456+0000","last_fullsized":"2024-03-08T08:29:07.409456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:08.239687+0000","last_change":"2024-03-08T08:29:05.251705+0000","last_active":"2024-03-08T08:29:08.239687+0000","last_peered":"2024-03-08T08:29:08.239687+0000","last_clean":"2024-03-08T08:29:08.239687+0000","last_became_active":"2024-03-08T08:29:05.251424+0000","last_became_peered":"2024-03-08T08:29:05.251424+0000","last_unstale":"2024-03-08T08:29:08.239687+0000","last_undegraded":"2024-03-08T08:29:08.239687+0000","last_fullsized":"2024-03-08T08:29:08.239687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235787+0000","last_change":"2024-03-08T08:29:06.248049+0000","last_active":"2024-03-08T08:29:07.235787+0000","last_peered":"2024-03-08T08:29:07.235787+0000","last_clean":"2024-03-08T08:29:07.235787+0000","last_became_active":"2024-03-08T08:29:06.243640+0000","last_became_peered":"2024-03-08T08:29:06.243640+0000","last_unstale":"2024-03-08T08:29:07.235787+0000","last_undegraded":"2024-03-08T08:29:07.235787+0000","last_fullsized":"2024-03-08T08:29:07.235787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:18:25.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995363+0000","last_change":"2024-03-08T08:29:05.256862+0000","last_active":"2024-03-08T08:29:09.995363+0000","last_peered":"2024-03-08T08:29:09.995363+0000","last_clean":"2024-03-08T08:29:09.995363+0000","last_became_active":"2024-03-08T08:29:05.256474+0000","last_became_peered":"2024-03-08T08:29:05.256474+0000","last_unstale":"2024-03-08T08:29:09.995363+0000","last_undegraded":"2024-03-08T08:29:09.995363+0000","last_fullsized":"2024-03-08T08:29:09.995363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235404+0000","last_change":"2024-03-08T08:29:06.240193+0000","last_active":"2024-03-08T08:29:07.235404+0000","last_peered":"2024-03-08T08:29:07.235404+0000","last_clean":"2024-03-08T08:29:07.235404+0000","last_became_active":"2024-03-08T08:29:06.239901+0000","last_became_peered":"2024-03-08T08:29:06.239901+0000","last_unstale":"2024-03-08T08:29:07.235404+0000","last_undegraded":"2024-03-08T08:29:07.235404+0000","last_fullsized":"2024-03-08T08:29:07.235404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562137+0000","last_change":"2024-03-08T08:29:05.250815+0000","last_active":"2024-03-08T08:29:07.562137+0000","last_peered":"2024-03-08T08:29:07.562137+0000","last_clean":"2024-03-08T08:29:07.562137+0000","last_became_active":"2024-03-08T08:29:05.250443+0000","last_became_peered":"2024-03-08T08:29:05.250443+0000","last_unstale":"2024-03-08T08:29:07.562137+0000","last_undegraded":"2024-03-08T08:29:07.562137+0000","last_fullsized":"2024-03-08T08:29:07.562137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995135+0000","last_change":"2024-03-08T08:29:06.253615+0000","last_active":"2024-03-08T08:29:09.995135+0000","last_peered":"2024-03-08T08:29:09.995135+0000","last_clean":"2024-03-08T08:29:09.995135+0000","last_became_active":"2024-03-08T08:29:06.253324+0000","last_became_peered":"2024-03-08T08:29:06.253324+0000","last_unstale":"2024-03-08T08:29:09.995135+0000","last_undegraded":"2024-03-08T08:29:09.995135+0000","last_fullsized":"2024-03-08T08:29:09.995135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:47:29.808023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.251684+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.251395+0000","last_became_peered":"2024-03-08T08:29:05.251395+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235017+0000","last_change":"2024-03-08T08:29:06.237785+0000","last_active":"2024-03-08T08:29:07.235017+0000","last_peered":"2024-03-08T08:29:07.235017+0000","last_clean":"2024-03-08T08:29:07.235017+0000","last_became_active":"2024-03-08T08:29:06.237449+0000","last_became_peered":"2024-03-08T08:29:06.237449+0000","last_unstale":"2024-03-08T08:29:07.235017+0000","last_undegraded":"2024-03-08T08:29:07.235017+0000","last_fullsized":"2024-03-08T08:29:07.235017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409279+0000","last_change":"2024-03-08T08:29:05.246589+0000","last_active":"2024-03-08T08:29:07.409279+0000","last_peered":"2024-03-08T08:29:07.409279+0000","last_clean":"2024-03-08T08:29:07.409279+0000","last_became_active":"2024-03-08T08:29:05.246204+0000","last_became_peered":"2024-03-08T08:29:05.246204+0000","last_unstale":"2024-03-08T08:29:07.409279+0000","last_undegraded":"2024-03-08T08:29:07.409279+0000","last_fullsized":"2024-03-08T08:29:07.409279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235180+0000","last_change":"2024-03-08T08:29:06.244533+0000","last_active":"2024-03-08T08:29:07.235180+0000","last_peered":"2024-03-08T08:29:07.235180+0000","last_clean":"2024-03-08T08:29:07.235180+0000","last_became_active":"2024-03-08T08:29:06.244232+0000","last_became_peered":"2024-03-08T08:29:06.244232+0000","last_unstale":"2024-03-08T08:29:07.235180+0000","last_undegraded":"2024-03-08T08:29:07.235180+0000","last_fullsized":"2024-03-08T08:29:07.235180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:36.028927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994933+0000","last_change":"2024-03-08T08:29:05.258093+0000","last_active":"2024-03-08T08:29:09.994933+0000","last_peered":"2024-03-08T08:29:09.994933+0000","last_clean":"2024-03-08T08:29:09.994933+0000","last_became_active":"2024-03-08T08:29:05.257817+0000","last_became_peered":"2024-03-08T08:29:05.257817+0000","last_unstale":"2024-03-08T08:29:09.994933+0000","last_undegraded":"2024-03-08T08:29:09.994933+0000","last_fullsized":"2024-03-08T08:29:09.994933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:18:30.428380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995344+0000","last_change":"2024-03-08T08:29:06.258011+0000","last_active":"2024-03-08T08:29:09.995344+0000","last_peered":"2024-03-08T08:29:09.995344+0000","last_clean":"2024-03-08T08:29:09.995344+0000","last_became_active":"2024-03-08T08:29:06.257742+0000","last_became_peered":"2024-03-08T08:29:06.257742+0000","last_unstale":"2024-03-08T08:29:09.995344+0000","last_undegraded":"2024-03-08T08:29:09.995344+0000","last_fullsized":"2024-03-08T08:29:09.995344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:05:29.450798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995182+0000","last_change":"2024-03-08T08:29:06.262040+0000","last_active":"2024-03-08T08:29:09.995182+0000","last_peered":"2024-03-08T08:29:09.995182+0000","last_clean":"2024-03-08T08:29:09.995182+0000","last_became_active":"2024-03-08T08:29:06.258293+0000","last_became_peered":"2024-03-08T08:29:06.258293+0000","last_unstale":"2024-03-08T08:29:09.995182+0000","last_undegraded":"2024-03-08T08:29:09.995182+0000","last_fullsized":"2024-03-08T08:29:09.995182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:36:28.954776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562776+0000","last_change":"2024-03-08T08:29:05.252744+0000","last_active":"2024-03-08T08:29:07.562776+0000","last_peered":"2024-03-08T08:29:07.562776+0000","last_clean":"2024-03-08T08:29:07.562776+0000","last_became_active":"2024-03-08T08:29:05.252365+0000","last_became_peered":"2024-03-08T08:29:05.252365+0000","last_unstale":"2024-03-08T08:29:07.562776+0000","last_undegraded":"2024-03-08T08:29:07.562776+0000","last_fullsized":"2024-03-08T08:29:07.562776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236776+0000","last_change":"2024-03-08T08:29:06.260541+0000","last_active":"2024-03-08T08:29:07.236776+0000","last_peered":"2024-03-08T08:29:07.236776+0000","last_clean":"2024-03-08T08:29:07.236776+0000","last_became_active":"2024-03-08T08:29:06.260283+0000","last_became_peered":"2024-03-08T08:29:06.260283+0000","last_unstale":"2024-03-08T08:29:07.236776+0000","last_undegraded":"2024-03-08T08:29:07.236776+0000","last_fullsized":"2024-03-08T08:29:07.236776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:52:33.000979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562608+0000","last_change":"2024-03-08T08:29:05.251622+0000","last_active":"2024-03-08T08:29:07.562608+0000","last_peered":"2024-03-08T08:29:07.562608+0000","last_clean":"2024-03-08T08:29:07.562608+0000","last_became_active":"2024-03-08T08:29:05.251346+0000","last_became_peered":"2024-03-08T08:29:05.251346+0000","last_unstale":"2024-03-08T08:29:07.562608+0000","last_undegraded":"2024-03-08T08:29:07.562608+0000","last_fullsized":"2024-03-08T08:29:07.562608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905651+0000","last_change":"2024-03-08T08:29:06.245002+0000","last_active":"2024-03-08T08:29:07.905651+0000","last_peered":"2024-03-08T08:29:07.905651+0000","last_clean":"2024-03-08T08:29:07.905651+0000","last_became_active":"2024-03-08T08:29:06.244765+0000","last_became_peered":"2024-03-08T08:29:06.244765+0000","last_unstale":"2024-03-08T08:29:07.905651+0000","last_undegraded":"2024-03-08T08:29:07.905651+0000","last_fullsized":"2024-03-08T08:29:07.905651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:14:06.939392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905631+0000","last_change":"2024-03-08T08:29:05.245332+0000","last_active":"2024-03-08T08:29:07.905631+0000","last_peered":"2024-03-08T08:29:07.905631+0000","last_clean":"2024-03-08T08:29:07.905631+0000","last_became_active":"2024-03-08T08:29:05.245042+0000","last_became_peered":"2024-03-08T08:29:05.245042+0000","last_unstale":"2024-03-08T08:29:07.905631+0000","last_undegraded":"2024-03-08T08:29:07.905631+0000","last_fullsized":"2024-03-08T08:29:07.905631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:36:06.948887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905465+0000","last_change":"2024-03-08T08:29:06.251546+0000","last_active":"2024-03-08T08:29:07.905465+0000","last_peered":"2024-03-08T08:29:07.905465+0000","last_clean":"2024-03-08T08:29:07.905465+0000","last_became_active":"2024-03-08T08:29:06.251278+0000","last_became_peered":"2024-03-08T08:29:06.251278+0000","last_unstale":"2024-03-08T08:29:07.905465+0000","last_undegraded":"2024-03-08T08:29:07.905465+0000","last_fullsized":"2024-03-08T08:29:07.905465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:29:28.214850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235915+0000","last_change":"2024-03-08T08:29:05.253532+0000","last_active":"2024-03-08T08:29:07.235915+0000","last_peered":"2024-03-08T08:29:07.235915+0000","last_clean":"2024-03-08T08:29:07.235915+0000","last_became_active":"2024-03-08T08:29:05.252904+0000","last_became_peered":"2024-03-08T08:29:05.252904+0000","last_unstale":"2024-03-08T08:29:07.235915+0000","last_undegraded":"2024-03-08T08:29:07.235915+0000","last_fullsized":"2024-03-08T08:29:07.235915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":73883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36144,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335696,"statfs":{"total":96636764160,"available":96599752704,"internally_reserved":0,"allocated":815104,"data_stored":664343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":76167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:18.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:29:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.240+0000 7f1568f5e640 1 Processor -- start 2024-03-08T08:29:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.240+0000 7f1568f5e640 1 -- start start 2024-03-08T08:29:18.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1568f5e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15640d1110 0x7f15640b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1568f5e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f15640d31c0 0x7f15640b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1568f5e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156415e1a0 con 0x7f156415ddc0 2024-03-08T08:29:18.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1568f5e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640b9030 con 0x7f15640d1110 2024-03-08T08:29:18.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1568f5e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640d1d20 con 0x7f15640d31c0 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15640d1110 0x7f15640b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15640d1110 0x7f15640b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58198/0 (socket says 172.21.15.12:58198) 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f156259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f156415ddc0 0x7f15640d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52930/0 (socket says 172.21.15.12:52930) 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 -- 172.21.15.12:0/2215853171 learned_addr learned my addr 172.21.15.12:0/2215853171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1562d9d640 1 --2- 172.21.15.12:0/2215853171 >> v2:172.21.15.156:3300/0 conn(0x7f15640d31c0 0x7f15640b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 -- 172.21.15.12:0/2215853171 >> v2:172.21.15.156:3300/0 conn(0x7f15640d31c0 msgr2=0x7f15640b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 --2- 172.21.15.12:0/2215853171 >> v2:172.21.15.156:3300/0 conn(0x7f15640d31c0 0x7f15640b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 -- 172.21.15.12:0/2215853171 >> v1:172.21.15.12:6789/0 conn(0x7f156415ddc0 legacy=0x7f15640d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f155bfff640 1 -- 172.21.15.12:0/2215853171 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640b8790 con 0x7f15640d1110 2024-03-08T08:29:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.241+0000 7f1561d9b640 1 -- 172.21.15.12:0/2215853171 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659641021 0 0) 0x7f156415e1a0 con 0x7f156415ddc0 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f155bfff640 1 --2- 172.21.15.12:0/2215853171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d1110 0x7f15640b6be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f15640ce090 tx=0x7f154c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83326881b80ecc0d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1561d9b640 1 -- 172.21.15.12:0/2215853171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c034070 con 0x7f15640d1110 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1561d9b640 1 -- 172.21.15.12:0/2215853171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c030860 con 0x7f15640d1110 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1561d9b640 1 -- 172.21.15.12:0/2215853171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c030c30 con 0x7f15640d1110 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 -- 172.21.15.12:0/2215853171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d1110 msgr2=0x7f15640b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 --2- 172.21.15.12:0/2215853171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d1110 0x7f15640b6be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f15640ce090 tx=0x7f154c02b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 -- 172.21.15.12:0/2215853171 shutdown_connections 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 --2- 172.21.15.12:0/2215853171 >> v2:172.21.15.156:3300/0 conn(0x7f15640d31c0 0x7f15640b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 --2- 172.21.15.12:0/2215853171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d1110 0x7f15640b6be0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.242+0000 7f1568f5e640 1 -- 172.21.15.12:0/2215853171 >> 172.21.15.12:0/2215853171 conn(0x7f1564072720 msgr2=0x7f1564072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.243+0000 7f1568f5e640 1 -- 172.21.15.12:0/2215853171 shutdown_connections 2024-03-08T08:29:18.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.243+0000 7f1568f5e640 1 -- 172.21.15.12:0/2215853171 wait complete. 2024-03-08T08:29:18.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 Processor -- start 2024-03-08T08:29:18.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 -- start start 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 0x7f15640cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f156415ddc0 0x7f156415fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b9980 con 0x7f15640d31c0 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b8d50 con 0x7f15640d1110 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.244+0000 7f1568f5e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b7660 con 0x7f156415ddc0 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58204/0 (socket says 172.21.15.12:58204) 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 -- 172.21.15.12:0/1978055366 learned_addr learned my addr 172.21.15.12:0/1978055366 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f156259c640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 0x7f15640cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f1562d9d640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f156415ddc0 0x7f156415fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 -- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f156415ddc0 msgr2=0x7f156415fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f156415ddc0 0x7f156415fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 -- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 msgr2=0x7f15640cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 0x7f15640cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 -- 172.21.15.12:0/1978055366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15641f4090 con 0x7f15640d31c0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f156259c640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 0x7f15640cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155bfff640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f154c02b9e0 tx=0x7f154c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c048070 con 0x7f15640d31c0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c030660 con 0x7f15640d31c0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15641f4370 con 0x7f15640d31c0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c034040 con 0x7f15640d31c0 2024-03-08T08:29:18.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.245+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156415f480 con 0x7f15640d31c0 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.246+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15640749c0 con 0x7f15640d31c0 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.246+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f154c04b070 con 0x7f15640d31c0 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.247+0000 7f155affd640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.247+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f154c0ac210 con 0x7f15640d31c0 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.247+0000 7f156259c640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.247+0000 7f156259c640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 0x7f154406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f60 tx=0x7f1554009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.251+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f154c07b090 con 0x7f15640d31c0 2024-03-08T08:29:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.427+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15640b5310 con 0x7f154406b4d0 2024-03-08T08:29:18.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.433+0000 7f155affd640 1 -- 172.21.15.12:0/1978055366 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251833 (secure 0 0 0) 0x7f15640b5310 con 0x7f154406b4d0 2024-03-08T08:29:18.437 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 msgr2=0x7f154406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 0x7f154406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f60 tx=0x7f1554009040 comp rx=0 tx=0).stop 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 msgr2=0x7f15641f3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f154c02b9e0 tx=0x7f154c004730 comp rx=0 tx=0).stop 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 shutdown_connections 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f156415ddc0 0x7f156415fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15640d31c0 0x7f15641f3720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 --2- 172.21.15.12:0/1978055366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15640d1110 0x7f15640cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 >> 172.21.15.12:0/1978055366 conn(0x7f1564072720 msgr2=0x7f15640d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 shutdown_connections 2024-03-08T08:29:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.436+0000 7f1568f5e640 1 -- 172.21.15.12:0/1978055366 wait complete. 2024-03-08T08:29:18.448 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-08T08:29:16.995168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239880,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734840,"statfs":{"total":773094113280,"available":772848476160,"internally_reserved":0,"allocated":2957312,"data_stored":1776620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002248"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562754+0000","last_change":"2024-03-08T08:29:06.251346+0000","last_active":"2024-03-08T08:29:07.562754+0000","last_peered":"2024-03-08T08:29:07.562754+0000","last_clean":"2024-03-08T08:29:07.562754+0000","last_became_active":"2024-03-08T08:29:06.250986+0000","last_became_peered":"2024-03-08T08:29:06.250986+0000","last_unstale":"2024-03-08T08:29:07.562754+0000","last_undegraded":"2024-03-08T08:29:07.562754+0000","last_fullsized":"2024-03-08T08:29:07.562754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:31:13.095757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905464+0000","last_change":"2024-03-08T08:29:06.247932+0000","last_active":"2024-03-08T08:29:07.905464+0000","last_peered":"2024-03-08T08:29:07.905464+0000","last_clean":"2024-03-08T08:29:07.905464+0000","last_became_active":"2024-03-08T08:29:06.247623+0000","last_became_peered":"2024-03-08T08:29:06.247623+0000","last_unstale":"2024-03-08T08:29:07.905464+0000","last_undegraded":"2024-03-08T08:29:07.905464+0000","last_fullsized":"2024-03-08T08:29:07.905464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:48:05.979625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905335+0000","last_change":"2024-03-08T08:29:06.244061+0000","last_active":"2024-03-08T08:29:07.905335+0000","last_peered":"2024-03-08T08:29:07.905335+0000","last_clean":"2024-03-08T08:29:07.905335+0000","last_became_active":"2024-03-08T08:29:06.243801+0000","last_became_peered":"2024-03-08T08:29:06.243801+0000","last_unstale":"2024-03-08T08:29:07.905335+0000","last_undegraded":"2024-03-08T08:29:07.905335+0000","last_fullsized":"2024-03-08T08:29:07.905335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:49:40.562208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995367+0000","last_change":"2024-03-08T08:29:06.250895+0000","last_active":"2024-03-08T08:29:09.995367+0000","last_peered":"2024-03-08T08:29:09.995367+0000","last_clean":"2024-03-08T08:29:09.995367+0000","last_became_active":"2024-03-08T08:29:06.250560+0000","last_became_peered":"2024-03-08T08:29:06.250560+0000","last_unstale":"2024-03-08T08:29:09.995367+0000","last_undegraded":"2024-03-08T08:29:09.995367+0000","last_fullsized":"2024-03-08T08:29:09.995367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:22:34.858690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562498+0000","last_change":"2024-03-08T08:29:06.249635+0000","last_active":"2024-03-08T08:29:07.562498+0000","last_peered":"2024-03-08T08:29:07.562498+0000","last_clean":"2024-03-08T08:29:07.562498+0000","last_became_active":"2024-03-08T08:29:06.249356+0000","last_became_peered":"2024-03-08T08:29:06.249356+0000","last_unstale":"2024-03-08T08:29:07.562498+0000","last_undegraded":"2024-03-08T08:29:07.562498+0000","last_fullsized":"2024-03-08T08:29:07.562498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:47:43.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994809+0000","last_change":"2024-03-08T08:29:06.249994+0000","last_active":"2024-03-08T08:29:09.994809+0000","last_peered":"2024-03-08T08:29:09.994809+0000","last_clean":"2024-03-08T08:29:09.994809+0000","last_became_active":"2024-03-08T08:29:06.249682+0000","last_became_peered":"2024-03-08T08:29:06.249682+0000","last_unstale":"2024-03-08T08:29:09.994809+0000","last_undegraded":"2024-03-08T08:29:09.994809+0000","last_fullsized":"2024-03-08T08:29:09.994809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:36.028927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562203+0000","last_change":"2024-03-08T08:29:06.237963+0000","last_active":"2024-03-08T08:29:07.562203+0000","last_peered":"2024-03-08T08:29:07.562203+0000","last_clean":"2024-03-08T08:29:07.562203+0000","last_became_active":"2024-03-08T08:29:06.237654+0000","last_became_peered":"2024-03-08T08:29:06.237654+0000","last_unstale":"2024-03-08T08:29:07.562203+0000","last_undegraded":"2024-03-08T08:29:07.562203+0000","last_fullsized":"2024-03-08T08:29:07.562203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:16:24.171714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905138+0000","last_change":"2024-03-08T08:29:06.258895+0000","last_active":"2024-03-08T08:29:07.905138+0000","last_peered":"2024-03-08T08:29:07.905138+0000","last_clean":"2024-03-08T08:29:07.905138+0000","last_became_active":"2024-03-08T08:29:06.258630+0000","last_became_peered":"2024-03-08T08:29:06.258630+0000","last_unstale":"2024-03-08T08:29:07.905138+0000","last_undegraded":"2024-03-08T08:29:07.905138+0000","last_fullsized":"2024-03-08T08:29:07.905138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:02:10.390227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236928+0000","last_change":"2024-03-08T08:29:06.260239+0000","last_active":"2024-03-08T08:29:07.236928+0000","last_peered":"2024-03-08T08:29:07.236928+0000","last_clean":"2024-03-08T08:29:07.236928+0000","last_became_active":"2024-03-08T08:29:06.259944+0000","last_became_peered":"2024-03-08T08:29:06.259944+0000","last_unstale":"2024-03-08T08:29:07.236928+0000","last_undegraded":"2024-03-08T08:29:07.236928+0000","last_fullsized":"2024-03-08T08:29:07.236928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:18:09.551413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562557+0000","last_change":"2024-03-08T08:29:06.252950+0000","last_active":"2024-03-08T08:29:07.562557+0000","last_peered":"2024-03-08T08:29:07.562557+0000","last_clean":"2024-03-08T08:29:07.562557+0000","last_became_active":"2024-03-08T08:29:06.252480+0000","last_became_peered":"2024-03-08T08:29:06.252480+0000","last_unstale":"2024-03-08T08:29:07.562557+0000","last_undegraded":"2024-03-08T08:29:07.562557+0000","last_fullsized":"2024-03-08T08:29:07.562557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:40:11.278068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562751+0000","last_change":"2024-03-08T08:29:06.248718+0000","last_active":"2024-03-08T08:29:07.562751+0000","last_peered":"2024-03-08T08:29:07.562751+0000","last_clean":"2024-03-08T08:29:07.562751+0000","last_became_active":"2024-03-08T08:29:06.248400+0000","last_became_peered":"2024-03-08T08:29:06.248400+0000","last_unstale":"2024-03-08T08:29:07.562751+0000","last_undegraded":"2024-03-08T08:29:07.562751+0000","last_fullsized":"2024-03-08T08:29:07.562751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:02:07.589132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562439+0000","last_change":"2024-03-08T08:29:06.247528+0000","last_active":"2024-03-08T08:29:07.562439+0000","last_peered":"2024-03-08T08:29:07.562439+0000","last_clean":"2024-03-08T08:29:07.562439+0000","last_became_active":"2024-03-08T08:29:06.246926+0000","last_became_peered":"2024-03-08T08:29:06.246926+0000","last_unstale":"2024-03-08T08:29:07.562439+0000","last_undegraded":"2024-03-08T08:29:07.562439+0000","last_fullsized":"2024-03-08T08:29:07.562439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:42.233464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409364+0000","last_change":"2024-03-08T08:29:06.261811+0000","last_active":"2024-03-08T08:29:07.409364+0000","last_peered":"2024-03-08T08:29:07.409364+0000","last_clean":"2024-03-08T08:29:07.409364+0000","last_became_active":"2024-03-08T08:29:06.261568+0000","last_became_peered":"2024-03-08T08:29:06.261568+0000","last_unstale":"2024-03-08T08:29:07.409364+0000","last_undegraded":"2024-03-08T08:29:07.409364+0000","last_fullsized":"2024-03-08T08:29:07.409364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409154+0000","last_change":"2024-03-08T08:29:06.245702+0000","last_active":"2024-03-08T08:29:07.409154+0000","last_peered":"2024-03-08T08:29:07.409154+0000","last_clean":"2024-03-08T08:29:07.409154+0000","last_became_active":"2024-03-08T08:29:06.245425+0000","last_became_peered":"2024-03-08T08:29:06.245425+0000","last_unstale":"2024-03-08T08:29:07.409154+0000","last_undegraded":"2024-03-08T08:29:07.409154+0000","last_fullsized":"2024-03-08T08:29:07.409154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:01.699386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905113+0000","last_change":"2024-03-08T08:29:06.250908+0000","last_active":"2024-03-08T08:29:07.905113+0000","last_peered":"2024-03-08T08:29:07.905113+0000","last_clean":"2024-03-08T08:29:07.905113+0000","last_became_active":"2024-03-08T08:29:06.250627+0000","last_became_peered":"2024-03-08T08:29:06.250627+0000","last_unstale":"2024-03-08T08:29:07.905113+0000","last_undegraded":"2024-03-08T08:29:07.905113+0000","last_fullsized":"2024-03-08T08:29:07.905113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:26:00.214895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994982+0000","last_change":"2024-03-08T08:29:06.257106+0000","last_active":"2024-03-08T08:29:09.994982+0000","last_peered":"2024-03-08T08:29:09.994982+0000","last_clean":"2024-03-08T08:29:09.994982+0000","last_became_active":"2024-03-08T08:29:06.256817+0000","last_became_peered":"2024-03-08T08:29:06.256817+0000","last_unstale":"2024-03-08T08:29:09.994982+0000","last_undegraded":"2024-03-08T08:29:09.994982+0000","last_fullsized":"2024-03-08T08:29:09.994982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:02:07.589132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995264+0000","last_change":"2024-03-08T08:29:06.262604+0000","last_active":"2024-03-08T08:29:09.995264+0000","last_peered":"2024-03-08T08:29:09.995264+0000","last_clean":"2024-03-08T08:29:09.995264+0000","last_became_active":"2024-03-08T08:29:06.262222+0000","last_became_peered":"2024-03-08T08:29:06.262222+0000","last_unstale":"2024-03-08T08:29:09.995264+0000","last_undegraded":"2024-03-08T08:29:09.995264+0000","last_fullsized":"2024-03-08T08:29:09.995264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:05:14.678904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235999+0000","last_change":"2024-03-08T08:29:06.256729+0000","last_active":"2024-03-08T08:29:07.235999+0000","last_peered":"2024-03-08T08:29:07.235999+0000","last_clean":"2024-03-08T08:29:07.235999+0000","last_became_active":"2024-03-08T08:29:06.256220+0000","last_became_peered":"2024-03-08T08:29:06.256220+0000","last_unstale":"2024-03-08T08:29:07.235999+0000","last_undegraded":"2024-03-08T08:29:07.235999+0000","last_fullsized":"2024-03-08T08:29:07.235999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:16.445063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995649+0000","last_change":"2024-03-08T08:29:06.251913+0000","last_active":"2024-03-08T08:29:09.995649+0000","last_peered":"2024-03-08T08:29:09.995649+0000","last_clean":"2024-03-08T08:29:09.995649+0000","last_became_active":"2024-03-08T08:29:06.251609+0000","last_became_peered":"2024-03-08T08:29:06.251609+0000","last_unstale":"2024-03-08T08:29:09.995649+0000","last_undegraded":"2024-03-08T08:29:09.995649+0000","last_fullsized":"2024-03-08T08:29:09.995649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:17:00.254942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562581+0000","last_change":"2024-03-08T08:29:06.247738+0000","last_active":"2024-03-08T08:29:07.562581+0000","last_peered":"2024-03-08T08:29:07.562581+0000","last_clean":"2024-03-08T08:29:07.562581+0000","last_became_active":"2024-03-08T08:29:06.247254+0000","last_became_peered":"2024-03-08T08:29:06.247254+0000","last_unstale":"2024-03-08T08:29:07.562581+0000","last_undegraded":"2024-03-08T08:29:07.562581+0000","last_fullsized":"2024-03-08T08:29:07.562581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:01:26.719267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235275+0000","last_change":"2024-03-08T08:29:06.251015+0000","last_active":"2024-03-08T08:29:07.235275+0000","last_peered":"2024-03-08T08:29:07.235275+0000","last_clean":"2024-03-08T08:29:07.235275+0000","last_became_active":"2024-03-08T08:29:06.250716+0000","last_became_peered":"2024-03-08T08:29:06.250716+0000","last_unstale":"2024-03-08T08:29:07.235275+0000","last_undegraded":"2024-03-08T08:29:07.235275+0000","last_fullsized":"2024-03-08T08:29:07.235275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236632+0000","last_change":"2024-03-08T08:29:06.259314+0000","last_active":"2024-03-08T08:29:07.236632+0000","last_peered":"2024-03-08T08:29:07.236632+0000","last_clean":"2024-03-08T08:29:07.236632+0000","last_became_active":"2024-03-08T08:29:06.258967+0000","last_became_peered":"2024-03-08T08:29:06.258967+0000","last_unstale":"2024-03-08T08:29:07.236632+0000","last_undegraded":"2024-03-08T08:29:07.236632+0000","last_fullsized":"2024-03-08T08:29:07.236632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:04:34.787442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905406+0000","last_change":"2024-03-08T08:29:06.251496+0000","last_active":"2024-03-08T08:29:07.905406+0000","last_peered":"2024-03-08T08:29:07.905406+0000","last_clean":"2024-03-08T08:29:07.905406+0000","last_became_active":"2024-03-08T08:29:06.251220+0000","last_became_peered":"2024-03-08T08:29:06.251220+0000","last_unstale":"2024-03-08T08:29:07.905406+0000","last_undegraded":"2024-03-08T08:29:07.905406+0000","last_fullsized":"2024-03-08T08:29:07.905406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:31:41.063261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994650+0000","last_change":"2024-03-08T08:29:06.256404+0000","last_active":"2024-03-08T08:29:09.994650+0000","last_peered":"2024-03-08T08:29:09.994650+0000","last_clean":"2024-03-08T08:29:09.994650+0000","last_became_active":"2024-03-08T08:29:06.256087+0000","last_became_peered":"2024-03-08T08:29:06.256087+0000","last_unstale":"2024-03-08T08:29:09.994650+0000","last_undegraded":"2024-03-08T08:29:09.994650+0000","last_fullsized":"2024-03-08T08:29:09.994650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:01:26.719267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905562+0000","last_change":"2024-03-08T08:29:06.255787+0000","last_active":"2024-03-08T08:29:07.905562+0000","last_peered":"2024-03-08T08:29:07.905562+0000","last_clean":"2024-03-08T08:29:07.905562+0000","last_became_active":"2024-03-08T08:29:06.255548+0000","last_became_peered":"2024-03-08T08:29:06.255548+0000","last_unstale":"2024-03-08T08:29:07.905562+0000","last_undegraded":"2024-03-08T08:29:07.905562+0000","last_fullsized":"2024-03-08T08:29:07.905562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:28:26.547413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995638+0000","last_change":"2024-03-08T08:29:06.259612+0000","last_active":"2024-03-08T08:29:09.995638+0000","last_peered":"2024-03-08T08:29:09.995638+0000","last_clean":"2024-03-08T08:29:09.995638+0000","last_became_active":"2024-03-08T08:29:06.259198+0000","last_became_peered":"2024-03-08T08:29:06.259198+0000","last_unstale":"2024-03-08T08:29:09.995638+0000","last_undegraded":"2024-03-08T08:29:09.995638+0000","last_fullsized":"2024-03-08T08:29:09.995638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:05.929603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409477+0000","last_change":"2024-03-08T08:29:06.245080+0000","last_active":"2024-03-08T08:29:07.409477+0000","last_peered":"2024-03-08T08:29:07.409477+0000","last_clean":"2024-03-08T08:29:07.409477+0000","last_became_active":"2024-03-08T08:29:06.244821+0000","last_became_peered":"2024-03-08T08:29:06.244821+0000","last_unstale":"2024-03-08T08:29:07.409477+0000","last_undegraded":"2024-03-08T08:29:07.409477+0000","last_fullsized":"2024-03-08T08:29:07.409477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409429+0000","last_change":"2024-03-08T08:29:06.259724+0000","last_active":"2024-03-08T08:29:07.409429+0000","last_peered":"2024-03-08T08:29:07.409429+0000","last_clean":"2024-03-08T08:29:07.409429+0000","last_became_active":"2024-03-08T08:29:06.259415+0000","last_became_peered":"2024-03-08T08:29:06.259415+0000","last_unstale":"2024-03-08T08:29:07.409429+0000","last_undegraded":"2024-03-08T08:29:07.409429+0000","last_fullsized":"2024-03-08T08:29:07.409429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:15:30.388731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905249+0000","last_change":"2024-03-08T08:29:06.248138+0000","last_active":"2024-03-08T08:29:07.905249+0000","last_peered":"2024-03-08T08:29:07.905249+0000","last_clean":"2024-03-08T08:29:07.905249+0000","last_became_active":"2024-03-08T08:29:06.247330+0000","last_became_peered":"2024-03-08T08:29:06.247330+0000","last_unstale":"2024-03-08T08:29:07.905249+0000","last_undegraded":"2024-03-08T08:29:07.905249+0000","last_fullsized":"2024-03-08T08:29:07.905249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:40:15.729074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995284+0000","last_change":"2024-03-08T08:29:06.254998+0000","last_active":"2024-03-08T08:29:09.995284+0000","last_peered":"2024-03-08T08:29:09.995284+0000","last_clean":"2024-03-08T08:29:09.995284+0000","last_became_active":"2024-03-08T08:29:06.254596+0000","last_became_peered":"2024-03-08T08:29:06.254596+0000","last_unstale":"2024-03-08T08:29:09.995284+0000","last_undegraded":"2024-03-08T08:29:09.995284+0000","last_fullsized":"2024-03-08T08:29:09.995284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:04.577609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235098+0000","last_change":"2024-03-08T08:29:06.244995+0000","last_active":"2024-03-08T08:29:07.235098+0000","last_peered":"2024-03-08T08:29:07.235098+0000","last_clean":"2024-03-08T08:29:07.235098+0000","last_became_active":"2024-03-08T08:29:06.244712+0000","last_became_peered":"2024-03-08T08:29:06.244712+0000","last_unstale":"2024-03-08T08:29:07.235098+0000","last_undegraded":"2024-03-08T08:29:07.235098+0000","last_fullsized":"2024-03-08T08:29:07.235098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:23.178032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994651+0000","last_change":"2024-03-08T08:29:06.255239+0000","last_active":"2024-03-08T08:29:09.994651+0000","last_peered":"2024-03-08T08:29:09.994651+0000","last_clean":"2024-03-08T08:29:09.994651+0000","last_became_active":"2024-03-08T08:29:06.254635+0000","last_became_peered":"2024-03-08T08:29:06.254635+0000","last_unstale":"2024-03-08T08:29:09.994651+0000","last_undegraded":"2024-03-08T08:29:09.994651+0000","last_fullsized":"2024-03-08T08:29:09.994651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:23.178032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236746+0000","last_change":"2024-03-08T08:29:06.252849+0000","last_active":"2024-03-08T08:29:07.236746+0000","last_peered":"2024-03-08T08:29:07.236746+0000","last_clean":"2024-03-08T08:29:07.236746+0000","last_became_active":"2024-03-08T08:29:06.252559+0000","last_became_peered":"2024-03-08T08:29:06.252559+0000","last_unstale":"2024-03-08T08:29:07.236746+0000","last_undegraded":"2024-03-08T08:29:07.236746+0000","last_fullsized":"2024-03-08T08:29:07.236746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:06:45.449613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905459+0000","last_change":"2024-03-08T08:29:05.247749+0000","last_active":"2024-03-08T08:29:07.905459+0000","last_peered":"2024-03-08T08:29:07.905459+0000","last_clean":"2024-03-08T08:29:07.905459+0000","last_became_active":"2024-03-08T08:29:05.247492+0000","last_became_peered":"2024-03-08T08:29:05.247492+0000","last_unstale":"2024-03-08T08:29:07.905459+0000","last_undegraded":"2024-03-08T08:29:07.905459+0000","last_fullsized":"2024-03-08T08:29:07.905459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:14:48.392940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235447+0000","last_change":"2024-03-08T08:29:06.248631+0000","last_active":"2024-03-08T08:29:07.235447+0000","last_peered":"2024-03-08T08:29:07.235447+0000","last_clean":"2024-03-08T08:29:07.235447+0000","last_became_active":"2024-03-08T08:29:06.248337+0000","last_became_peered":"2024-03-08T08:29:06.248337+0000","last_unstale":"2024-03-08T08:29:07.235447+0000","last_undegraded":"2024-03-08T08:29:07.235447+0000","last_fullsized":"2024-03-08T08:29:07.235447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:41:43.373645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995383+0000","last_change":"2024-03-08T08:29:05.256580+0000","last_active":"2024-03-08T08:29:09.995383+0000","last_peered":"2024-03-08T08:29:09.995383+0000","last_clean":"2024-03-08T08:29:09.995383+0000","last_became_active":"2024-03-08T08:29:05.256286+0000","last_became_peered":"2024-03-08T08:29:05.256286+0000","last_unstale":"2024-03-08T08:29:09.995383+0000","last_undegraded":"2024-03-08T08:29:09.995383+0000","last_fullsized":"2024-03-08T08:29:09.995383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409076+0000","last_change":"2024-03-08T08:29:05.253153+0000","last_active":"2024-03-08T08:29:07.409076+0000","last_peered":"2024-03-08T08:29:07.409076+0000","last_clean":"2024-03-08T08:29:07.409076+0000","last_became_active":"2024-03-08T08:29:05.252819+0000","last_became_peered":"2024-03-08T08:29:05.252819+0000","last_unstale":"2024-03-08T08:29:07.409076+0000","last_undegraded":"2024-03-08T08:29:07.409076+0000","last_fullsized":"2024-03-08T08:29:07.409076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905491+0000","last_change":"2024-03-08T08:29:06.249692+0000","last_active":"2024-03-08T08:29:07.905491+0000","last_peered":"2024-03-08T08:29:07.905491+0000","last_clean":"2024-03-08T08:29:07.905491+0000","last_became_active":"2024-03-08T08:29:06.248953+0000","last_became_peered":"2024-03-08T08:29:06.248953+0000","last_unstale":"2024-03-08T08:29:07.905491+0000","last_undegraded":"2024-03-08T08:29:07.905491+0000","last_fullsized":"2024-03-08T08:29:07.905491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:30:01.675268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994744+0000","last_change":"2024-03-08T08:29:05.257702+0000","last_active":"2024-03-08T08:29:09.994744+0000","last_peered":"2024-03-08T08:29:09.994744+0000","last_clean":"2024-03-08T08:29:09.994744+0000","last_became_active":"2024-03-08T08:29:05.257406+0000","last_became_peered":"2024-03-08T08:29:05.257406+0000","last_unstale":"2024-03-08T08:29:09.994744+0000","last_undegraded":"2024-03-08T08:29:09.994744+0000","last_fullsized":"2024-03-08T08:29:09.994744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:43.430990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235273+0000","last_change":"2024-03-08T08:29:06.246139+0000","last_active":"2024-03-08T08:29:07.235273+0000","last_peered":"2024-03-08T08:29:07.235273+0000","last_clean":"2024-03-08T08:29:07.235273+0000","last_became_active":"2024-03-08T08:29:06.245660+0000","last_became_peered":"2024-03-08T08:29:06.245660+0000","last_unstale":"2024-03-08T08:29:07.235273+0000","last_undegraded":"2024-03-08T08:29:07.235273+0000","last_fullsized":"2024-03-08T08:29:07.235273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:46.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236862+0000","last_change":"2024-03-08T08:29:05.240557+0000","last_active":"2024-03-08T08:29:07.236862+0000","last_peered":"2024-03-08T08:29:07.236862+0000","last_clean":"2024-03-08T08:29:07.236862+0000","last_became_active":"2024-03-08T08:29:05.240233+0000","last_became_peered":"2024-03-08T08:29:05.240233+0000","last_unstale":"2024-03-08T08:29:07.236862+0000","last_undegraded":"2024-03-08T08:29:07.236862+0000","last_fullsized":"2024-03-08T08:29:07.236862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995697+0000","last_change":"2024-03-08T08:29:06.258421+0000","last_active":"2024-03-08T08:29:09.995697+0000","last_peered":"2024-03-08T08:29:09.995697+0000","last_clean":"2024-03-08T08:29:09.995697+0000","last_became_active":"2024-03-08T08:29:06.258079+0000","last_became_peered":"2024-03-08T08:29:06.258079+0000","last_unstale":"2024-03-08T08:29:09.995697+0000","last_undegraded":"2024-03-08T08:29:09.995697+0000","last_fullsized":"2024-03-08T08:29:09.995697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:18:31.433508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562645+0000","last_change":"2024-03-08T08:29:05.251488+0000","last_active":"2024-03-08T08:29:07.562645+0000","last_peered":"2024-03-08T08:29:07.562645+0000","last_clean":"2024-03-08T08:29:07.562645+0000","last_became_active":"2024-03-08T08:29:05.251220+0000","last_became_peered":"2024-03-08T08:29:05.251220+0000","last_unstale":"2024-03-08T08:29:07.562645+0000","last_undegraded":"2024-03-08T08:29:07.562645+0000","last_fullsized":"2024-03-08T08:29:07.562645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905718+0000","last_change":"2024-03-08T08:29:06.257799+0000","last_active":"2024-03-08T08:29:07.905718+0000","last_peered":"2024-03-08T08:29:07.905718+0000","last_clean":"2024-03-08T08:29:07.905718+0000","last_became_active":"2024-03-08T08:29:06.257545+0000","last_became_peered":"2024-03-08T08:29:06.257545+0000","last_unstale":"2024-03-08T08:29:07.905718+0000","last_undegraded":"2024-03-08T08:29:07.905718+0000","last_fullsized":"2024-03-08T08:29:07.905718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:21:44.744902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236028+0000","last_change":"2024-03-08T08:29:05.240046+0000","last_active":"2024-03-08T08:29:07.236028+0000","last_peered":"2024-03-08T08:29:07.236028+0000","last_clean":"2024-03-08T08:29:07.236028+0000","last_became_active":"2024-03-08T08:29:05.239754+0000","last_became_peered":"2024-03-08T08:29:05.239754+0000","last_unstale":"2024-03-08T08:29:07.236028+0000","last_undegraded":"2024-03-08T08:29:07.236028+0000","last_fullsized":"2024-03-08T08:29:07.236028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:38:44.493111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994857+0000","last_change":"2024-03-08T08:29:06.258228+0000","last_active":"2024-03-08T08:29:09.994857+0000","last_peered":"2024-03-08T08:29:09.994857+0000","last_clean":"2024-03-08T08:29:09.994857+0000","last_became_active":"2024-03-08T08:29:06.256849+0000","last_became_peered":"2024-03-08T08:29:06.256849+0000","last_unstale":"2024-03-08T08:29:09.994857+0000","last_undegraded":"2024-03-08T08:29:09.994857+0000","last_fullsized":"2024-03-08T08:29:09.994857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:31:13.095757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995463+0000","last_change":"2024-03-08T08:29:05.246882+0000","last_active":"2024-03-08T08:29:09.995463+0000","last_peered":"2024-03-08T08:29:09.995463+0000","last_clean":"2024-03-08T08:29:09.995463+0000","last_became_active":"2024-03-08T08:29:05.240261+0000","last_became_peered":"2024-03-08T08:29:05.240261+0000","last_unstale":"2024-03-08T08:29:09.995463+0000","last_undegraded":"2024-03-08T08:29:09.995463+0000","last_fullsized":"2024-03-08T08:29:09.995463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:20:06.037325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562506+0000","last_change":"2024-03-08T08:29:06.246529+0000","last_active":"2024-03-08T08:29:07.562506+0000","last_peered":"2024-03-08T08:29:07.562506+0000","last_clean":"2024-03-08T08:29:07.562506+0000","last_became_active":"2024-03-08T08:29:06.246230+0000","last_became_peered":"2024-03-08T08:29:06.246230+0000","last_unstale":"2024-03-08T08:29:07.562506+0000","last_undegraded":"2024-03-08T08:29:07.562506+0000","last_fullsized":"2024-03-08T08:29:07.562506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:23:31.113352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235779+0000","last_change":"2024-03-08T08:29:05.251723+0000","last_active":"2024-03-08T08:29:07.235779+0000","last_peered":"2024-03-08T08:29:07.235779+0000","last_clean":"2024-03-08T08:29:07.235779+0000","last_became_active":"2024-03-08T08:29:05.251410+0000","last_became_peered":"2024-03-08T08:29:05.251410+0000","last_unstale":"2024-03-08T08:29:07.235779+0000","last_undegraded":"2024-03-08T08:29:07.235779+0000","last_fullsized":"2024-03-08T08:29:07.235779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409264+0000","last_change":"2024-03-08T08:29:06.249661+0000","last_active":"2024-03-08T08:29:07.409264+0000","last_peered":"2024-03-08T08:29:07.409264+0000","last_clean":"2024-03-08T08:29:07.409264+0000","last_became_active":"2024-03-08T08:29:06.249388+0000","last_became_peered":"2024-03-08T08:29:06.249388+0000","last_unstale":"2024-03-08T08:29:07.409264+0000","last_undegraded":"2024-03-08T08:29:07.409264+0000","last_fullsized":"2024-03-08T08:29:07.409264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995432+0000","last_change":"2024-03-08T08:29:05.250954+0000","last_active":"2024-03-08T08:29:09.995432+0000","last_peered":"2024-03-08T08:29:09.995432+0000","last_clean":"2024-03-08T08:29:09.995432+0000","last_became_active":"2024-03-08T08:29:05.250478+0000","last_became_peered":"2024-03-08T08:29:05.250478+0000","last_unstale":"2024-03-08T08:29:09.995432+0000","last_undegraded":"2024-03-08T08:29:09.995432+0000","last_fullsized":"2024-03-08T08:29:09.995432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:22:33.853562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994832+0000","last_change":"2024-03-08T08:29:06.263756+0000","last_active":"2024-03-08T08:29:09.994832+0000","last_peered":"2024-03-08T08:29:09.994832+0000","last_clean":"2024-03-08T08:29:09.994832+0000","last_became_active":"2024-03-08T08:29:06.263461+0000","last_became_peered":"2024-03-08T08:29:06.263461+0000","last_unstale":"2024-03-08T08:29:09.994832+0000","last_undegraded":"2024-03-08T08:29:09.994832+0000","last_fullsized":"2024-03-08T08:29:09.994832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:37:49.534588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562192+0000","last_change":"2024-03-08T08:29:05.253842+0000","last_active":"2024-03-08T08:29:07.562192+0000","last_peered":"2024-03-08T08:29:07.562192+0000","last_clean":"2024-03-08T08:29:07.562192+0000","last_became_active":"2024-03-08T08:29:05.253552+0000","last_became_peered":"2024-03-08T08:29:05.253552+0000","last_unstale":"2024-03-08T08:29:07.562192+0000","last_undegraded":"2024-03-08T08:29:07.562192+0000","last_fullsized":"2024-03-08T08:29:07.562192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236519+0000","last_change":"2024-03-08T08:29:06.259240+0000","last_active":"2024-03-08T08:29:07.236519+0000","last_peered":"2024-03-08T08:29:07.236519+0000","last_clean":"2024-03-08T08:29:07.236519+0000","last_became_active":"2024-03-08T08:29:06.257182+0000","last_became_peered":"2024-03-08T08:29:06.257182+0000","last_unstale":"2024-03-08T08:29:07.236519+0000","last_undegraded":"2024-03-08T08:29:07.236519+0000","last_fullsized":"2024-03-08T08:29:07.236519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:58:34.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235055+0000","last_change":"2024-03-08T08:29:05.237318+0000","last_active":"2024-03-08T08:29:07.235055+0000","last_peered":"2024-03-08T08:29:07.235055+0000","last_clean":"2024-03-08T08:29:07.235055+0000","last_became_active":"2024-03-08T08:29:05.236985+0000","last_became_peered":"2024-03-08T08:29:05.236985+0000","last_unstale":"2024-03-08T08:29:07.235055+0000","last_undegraded":"2024-03-08T08:29:07.235055+0000","last_fullsized":"2024-03-08T08:29:07.235055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995565+0000","last_change":"2024-03-08T08:29:06.256312+0000","last_active":"2024-03-08T08:29:09.995565+0000","last_peered":"2024-03-08T08:29:09.995565+0000","last_clean":"2024-03-08T08:29:09.995565+0000","last_became_active":"2024-03-08T08:29:06.256021+0000","last_became_peered":"2024-03-08T08:29:06.256021+0000","last_unstale":"2024-03-08T08:29:09.995565+0000","last_undegraded":"2024-03-08T08:29:09.995565+0000","last_fullsized":"2024-03-08T08:29:09.995565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:22.530837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562837+0000","last_change":"2024-03-08T08:29:05.253285+0000","last_active":"2024-03-08T08:29:07.562837+0000","last_peered":"2024-03-08T08:29:07.562837+0000","last_clean":"2024-03-08T08:29:07.562837+0000","last_became_active":"2024-03-08T08:29:05.253023+0000","last_became_peered":"2024-03-08T08:29:05.253023+0000","last_unstale":"2024-03-08T08:29:07.562837+0000","last_undegraded":"2024-03-08T08:29:07.562837+0000","last_fullsized":"2024-03-08T08:29:07.562837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995783+0000","last_change":"2024-03-08T08:29:06.263316+0000","last_active":"2024-03-08T08:29:09.995783+0000","last_peered":"2024-03-08T08:29:09.995783+0000","last_clean":"2024-03-08T08:29:09.995783+0000","last_became_active":"2024-03-08T08:29:06.263033+0000","last_became_peered":"2024-03-08T08:29:06.263033+0000","last_unstale":"2024-03-08T08:29:09.995783+0000","last_undegraded":"2024-03-08T08:29:09.995783+0000","last_fullsized":"2024-03-08T08:29:09.995783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:01.699386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.247430+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.247146+0000","last_became_peered":"2024-03-08T08:29:05.247146+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:58:04.715697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562783+0000","last_change":"2024-03-08T08:29:06.246687+0000","last_active":"2024-03-08T08:29:07.562783+0000","last_peered":"2024-03-08T08:29:07.562783+0000","last_clean":"2024-03-08T08:29:07.562783+0000","last_became_active":"2024-03-08T08:29:06.246421+0000","last_became_peered":"2024-03-08T08:29:06.246421+0000","last_unstale":"2024-03-08T08:29:07.562783+0000","last_undegraded":"2024-03-08T08:29:07.562783+0000","last_fullsized":"2024-03-08T08:29:07.562783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:47:29.808023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905309+0000","last_change":"2024-03-08T08:29:05.246842+0000","last_active":"2024-03-08T08:29:07.905309+0000","last_peered":"2024-03-08T08:29:07.905309+0000","last_clean":"2024-03-08T08:29:07.905309+0000","last_became_active":"2024-03-08T08:29:05.246589+0000","last_became_peered":"2024-03-08T08:29:05.246589+0000","last_unstale":"2024-03-08T08:29:07.905309+0000","last_undegraded":"2024-03-08T08:29:07.905309+0000","last_fullsized":"2024-03-08T08:29:07.905309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:34:13.125245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562749+0000","last_change":"2024-03-08T08:29:06.250869+0000","last_active":"2024-03-08T08:29:07.562749+0000","last_peered":"2024-03-08T08:29:07.562749+0000","last_clean":"2024-03-08T08:29:07.562749+0000","last_became_active":"2024-03-08T08:29:06.250598+0000","last_became_peered":"2024-03-08T08:29:06.250598+0000","last_unstale":"2024-03-08T08:29:07.562749+0000","last_undegraded":"2024-03-08T08:29:07.562749+0000","last_fullsized":"2024-03-08T08:29:07.562749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:37:49.534588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409554+0000","last_change":"2024-03-08T08:29:05.253164+0000","last_active":"2024-03-08T08:29:07.409554+0000","last_peered":"2024-03-08T08:29:07.409554+0000","last_clean":"2024-03-08T08:29:07.409554+0000","last_became_active":"2024-03-08T08:29:05.252833+0000","last_became_peered":"2024-03-08T08:29:05.252833+0000","last_unstale":"2024-03-08T08:29:07.409554+0000","last_undegraded":"2024-03-08T08:29:07.409554+0000","last_fullsized":"2024-03-08T08:29:07.409554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995324+0000","last_change":"2024-03-08T08:29:06.259015+0000","last_active":"2024-03-08T08:29:09.995324+0000","last_peered":"2024-03-08T08:29:09.995324+0000","last_clean":"2024-03-08T08:29:09.995324+0000","last_became_active":"2024-03-08T08:29:06.258632+0000","last_became_peered":"2024-03-08T08:29:06.258632+0000","last_unstale":"2024-03-08T08:29:09.995324+0000","last_undegraded":"2024-03-08T08:29:09.995324+0000","last_fullsized":"2024-03-08T08:29:09.995324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:29:38.770610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236450+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:29:07.236450+0000","last_peered":"2024-03-08T08:29:07.236450+0000","last_clean":"2024-03-08T08:29:07.236450+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:29:07.236450+0000","last_undegraded":"2024-03-08T08:29:07.236450+0000","last_fullsized":"2024-03-08T08:29:07.236450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995046+0000","last_change":"2024-03-08T08:29:05.257931+0000","last_active":"2024-03-08T08:29:09.995046+0000","last_peered":"2024-03-08T08:29:09.995046+0000","last_clean":"2024-03-08T08:29:09.995046+0000","last_became_active":"2024-03-08T08:29:05.257637+0000","last_became_peered":"2024-03-08T08:29:05.257637+0000","last_unstale":"2024-03-08T08:29:09.995046+0000","last_undegraded":"2024-03-08T08:29:09.995046+0000","last_fullsized":"2024-03-08T08:29:09.995046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:33:00.694258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235934+0000","last_change":"2024-03-08T08:29:06.251166+0000","last_active":"2024-03-08T08:29:07.235934+0000","last_peered":"2024-03-08T08:29:07.235934+0000","last_clean":"2024-03-08T08:29:07.235934+0000","last_became_active":"2024-03-08T08:29:06.250778+0000","last_became_peered":"2024-03-08T08:29:06.250778+0000","last_unstale":"2024-03-08T08:29:07.235934+0000","last_undegraded":"2024-03-08T08:29:07.235934+0000","last_fullsized":"2024-03-08T08:29:07.235934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:14:53.496011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994847+0000","last_change":"2024-03-08T08:29:05.251244+0000","last_active":"2024-03-08T08:29:09.994847+0000","last_peered":"2024-03-08T08:29:09.994847+0000","last_clean":"2024-03-08T08:29:09.994847+0000","last_became_active":"2024-03-08T08:29:05.250928+0000","last_became_peered":"2024-03-08T08:29:05.250928+0000","last_unstale":"2024-03-08T08:29:09.994847+0000","last_undegraded":"2024-03-08T08:29:09.994847+0000","last_fullsized":"2024-03-08T08:29:09.994847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:52.269276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409152+0000","last_change":"2024-03-08T08:29:06.255405+0000","last_active":"2024-03-08T08:29:07.409152+0000","last_peered":"2024-03-08T08:29:07.409152+0000","last_clean":"2024-03-08T08:29:07.409152+0000","last_became_active":"2024-03-08T08:29:06.255118+0000","last_became_peered":"2024-03-08T08:29:06.255118+0000","last_unstale":"2024-03-08T08:29:07.409152+0000","last_undegraded":"2024-03-08T08:29:07.409152+0000","last_fullsized":"2024-03-08T08:29:07.409152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:33:44.436118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235211+0000","last_change":"2024-03-08T08:29:05.245447+0000","last_active":"2024-03-08T08:29:07.235211+0000","last_peered":"2024-03-08T08:29:07.235211+0000","last_clean":"2024-03-08T08:29:07.235211+0000","last_became_active":"2024-03-08T08:29:05.245148+0000","last_became_peered":"2024-03-08T08:29:05.245148+0000","last_unstale":"2024-03-08T08:29:07.235211+0000","last_undegraded":"2024-03-08T08:29:07.235211+0000","last_fullsized":"2024-03-08T08:29:07.235211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905516+0000","last_change":"2024-03-08T08:29:06.248546+0000","last_active":"2024-03-08T08:29:07.905516+0000","last_peered":"2024-03-08T08:29:07.905516+0000","last_clean":"2024-03-08T08:29:07.905516+0000","last_became_active":"2024-03-08T08:29:06.248292+0000","last_became_peered":"2024-03-08T08:29:06.248292+0000","last_unstale":"2024-03-08T08:29:07.905516+0000","last_undegraded":"2024-03-08T08:29:07.905516+0000","last_fullsized":"2024-03-08T08:29:07.905516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:32:56.591618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995527+0000","last_change":"2024-03-08T08:29:05.250674+0000","last_active":"2024-03-08T08:29:09.995527+0000","last_peered":"2024-03-08T08:29:09.995527+0000","last_clean":"2024-03-08T08:29:09.995527+0000","last_became_active":"2024-03-08T08:29:05.250342+0000","last_became_peered":"2024-03-08T08:29:05.250342+0000","last_unstale":"2024-03-08T08:29:09.995527+0000","last_undegraded":"2024-03-08T08:29:09.995527+0000","last_fullsized":"2024-03-08T08:29:09.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:16:04.924475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562458+0000","last_change":"2024-03-08T08:29:06.252736+0000","last_active":"2024-03-08T08:29:07.562458+0000","last_peered":"2024-03-08T08:29:07.562458+0000","last_clean":"2024-03-08T08:29:07.562458+0000","last_became_active":"2024-03-08T08:29:06.251566+0000","last_became_peered":"2024-03-08T08:29:06.251566+0000","last_unstale":"2024-03-08T08:29:07.562458+0000","last_undegraded":"2024-03-08T08:29:07.562458+0000","last_fullsized":"2024-03-08T08:29:07.562458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:52:02.676611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905374+0000","last_change":"2024-03-08T08:29:05.245146+0000","last_active":"2024-03-08T08:29:07.905374+0000","last_peered":"2024-03-08T08:29:07.905374+0000","last_clean":"2024-03-08T08:29:07.905374+0000","last_became_active":"2024-03-08T08:29:05.244812+0000","last_became_peered":"2024-03-08T08:29:05.244812+0000","last_unstale":"2024-03-08T08:29:07.905374+0000","last_undegraded":"2024-03-08T08:29:07.905374+0000","last_fullsized":"2024-03-08T08:29:07.905374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:40:51.620757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994818+0000","last_change":"2024-03-08T08:29:06.254952+0000","last_active":"2024-03-08T08:29:09.994818+0000","last_peered":"2024-03-08T08:29:09.994818+0000","last_clean":"2024-03-08T08:29:09.994818+0000","last_became_active":"2024-03-08T08:29:06.254408+0000","last_became_peered":"2024-03-08T08:29:06.254408+0000","last_unstale":"2024-03-08T08:29:09.994818+0000","last_undegraded":"2024-03-08T08:29:09.994818+0000","last_fullsized":"2024-03-08T08:29:09.994818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:16.445063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995531+0000","last_change":"2024-03-08T08:29:05.256587+0000","last_active":"2024-03-08T08:29:09.995531+0000","last_peered":"2024-03-08T08:29:09.995531+0000","last_clean":"2024-03-08T08:29:09.995531+0000","last_became_active":"2024-03-08T08:29:05.256091+0000","last_became_peered":"2024-03-08T08:29:05.256091+0000","last_unstale":"2024-03-08T08:29:09.995531+0000","last_undegraded":"2024-03-08T08:29:09.995531+0000","last_fullsized":"2024-03-08T08:29:09.995531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409456+0000","last_change":"2024-03-08T08:29:06.250994+0000","last_active":"2024-03-08T08:29:07.409456+0000","last_peered":"2024-03-08T08:29:07.409456+0000","last_clean":"2024-03-08T08:29:07.409456+0000","last_became_active":"2024-03-08T08:29:06.250738+0000","last_became_peered":"2024-03-08T08:29:06.250738+0000","last_unstale":"2024-03-08T08:29:07.409456+0000","last_undegraded":"2024-03-08T08:29:07.409456+0000","last_fullsized":"2024-03-08T08:29:07.409456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:08.239687+0000","last_change":"2024-03-08T08:29:05.251705+0000","last_active":"2024-03-08T08:29:08.239687+0000","last_peered":"2024-03-08T08:29:08.239687+0000","last_clean":"2024-03-08T08:29:08.239687+0000","last_became_active":"2024-03-08T08:29:05.251424+0000","last_became_peered":"2024-03-08T08:29:05.251424+0000","last_unstale":"2024-03-08T08:29:08.239687+0000","last_undegraded":"2024-03-08T08:29:08.239687+0000","last_fullsized":"2024-03-08T08:29:08.239687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:28.311361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235787+0000","last_change":"2024-03-08T08:29:06.248049+0000","last_active":"2024-03-08T08:29:07.235787+0000","last_peered":"2024-03-08T08:29:07.235787+0000","last_clean":"2024-03-08T08:29:07.235787+0000","last_became_active":"2024-03-08T08:29:06.243640+0000","last_became_peered":"2024-03-08T08:29:06.243640+0000","last_unstale":"2024-03-08T08:29:07.235787+0000","last_undegraded":"2024-03-08T08:29:07.235787+0000","last_fullsized":"2024-03-08T08:29:07.235787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:18:25.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995363+0000","last_change":"2024-03-08T08:29:05.256862+0000","last_active":"2024-03-08T08:29:09.995363+0000","last_peered":"2024-03-08T08:29:09.995363+0000","last_clean":"2024-03-08T08:29:09.995363+0000","last_became_active":"2024-03-08T08:29:05.256474+0000","last_became_peered":"2024-03-08T08:29:05.256474+0000","last_unstale":"2024-03-08T08:29:09.995363+0000","last_undegraded":"2024-03-08T08:29:09.995363+0000","last_fullsized":"2024-03-08T08:29:09.995363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235404+0000","last_change":"2024-03-08T08:29:06.240193+0000","last_active":"2024-03-08T08:29:07.235404+0000","last_peered":"2024-03-08T08:29:07.235404+0000","last_clean":"2024-03-08T08:29:07.235404+0000","last_became_active":"2024-03-08T08:29:06.239901+0000","last_became_peered":"2024-03-08T08:29:06.239901+0000","last_unstale":"2024-03-08T08:29:07.235404+0000","last_undegraded":"2024-03-08T08:29:07.235404+0000","last_fullsized":"2024-03-08T08:29:07.235404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:16:21.451736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562137+0000","last_change":"2024-03-08T08:29:05.250815+0000","last_active":"2024-03-08T08:29:07.562137+0000","last_peered":"2024-03-08T08:29:07.562137+0000","last_clean":"2024-03-08T08:29:07.562137+0000","last_became_active":"2024-03-08T08:29:05.250443+0000","last_became_peered":"2024-03-08T08:29:05.250443+0000","last_unstale":"2024-03-08T08:29:07.562137+0000","last_undegraded":"2024-03-08T08:29:07.562137+0000","last_fullsized":"2024-03-08T08:29:07.562137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:57:38.989241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995135+0000","last_change":"2024-03-08T08:29:06.253615+0000","last_active":"2024-03-08T08:29:09.995135+0000","last_peered":"2024-03-08T08:29:09.995135+0000","last_clean":"2024-03-08T08:29:09.995135+0000","last_became_active":"2024-03-08T08:29:06.253324+0000","last_became_peered":"2024-03-08T08:29:06.253324+0000","last_unstale":"2024-03-08T08:29:09.995135+0000","last_undegraded":"2024-03-08T08:29:09.995135+0000","last_fullsized":"2024-03-08T08:29:09.995135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:47:29.808023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235569+0000","last_change":"2024-03-08T08:29:05.251684+0000","last_active":"2024-03-08T08:29:07.235569+0000","last_peered":"2024-03-08T08:29:07.235569+0000","last_clean":"2024-03-08T08:29:07.235569+0000","last_became_active":"2024-03-08T08:29:05.251395+0000","last_became_peered":"2024-03-08T08:29:05.251395+0000","last_unstale":"2024-03-08T08:29:07.235569+0000","last_undegraded":"2024-03-08T08:29:07.235569+0000","last_fullsized":"2024-03-08T08:29:07.235569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:48:22.172904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235017+0000","last_change":"2024-03-08T08:29:06.237785+0000","last_active":"2024-03-08T08:29:07.235017+0000","last_peered":"2024-03-08T08:29:07.235017+0000","last_clean":"2024-03-08T08:29:07.235017+0000","last_became_active":"2024-03-08T08:29:06.237449+0000","last_became_peered":"2024-03-08T08:29:06.237449+0000","last_unstale":"2024-03-08T08:29:07.235017+0000","last_undegraded":"2024-03-08T08:29:07.235017+0000","last_fullsized":"2024-03-08T08:29:07.235017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:35:53.274404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.409279+0000","last_change":"2024-03-08T08:29:05.246589+0000","last_active":"2024-03-08T08:29:07.409279+0000","last_peered":"2024-03-08T08:29:07.409279+0000","last_clean":"2024-03-08T08:29:07.409279+0000","last_became_active":"2024-03-08T08:29:05.246204+0000","last_became_peered":"2024-03-08T08:29:05.246204+0000","last_unstale":"2024-03-08T08:29:07.409279+0000","last_undegraded":"2024-03-08T08:29:07.409279+0000","last_fullsized":"2024-03-08T08:29:07.409279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:34:20.078674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235180+0000","last_change":"2024-03-08T08:29:06.244533+0000","last_active":"2024-03-08T08:29:07.235180+0000","last_peered":"2024-03-08T08:29:07.235180+0000","last_clean":"2024-03-08T08:29:07.235180+0000","last_became_active":"2024-03-08T08:29:06.244232+0000","last_became_peered":"2024-03-08T08:29:06.244232+0000","last_unstale":"2024-03-08T08:29:07.235180+0000","last_undegraded":"2024-03-08T08:29:07.235180+0000","last_fullsized":"2024-03-08T08:29:07.235180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:36.028927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.994933+0000","last_change":"2024-03-08T08:29:05.258093+0000","last_active":"2024-03-08T08:29:09.994933+0000","last_peered":"2024-03-08T08:29:09.994933+0000","last_clean":"2024-03-08T08:29:09.994933+0000","last_became_active":"2024-03-08T08:29:05.257817+0000","last_became_peered":"2024-03-08T08:29:05.257817+0000","last_unstale":"2024-03-08T08:29:09.994933+0000","last_undegraded":"2024-03-08T08:29:09.994933+0000","last_fullsized":"2024-03-08T08:29:09.994933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:18:30.428380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995344+0000","last_change":"2024-03-08T08:29:06.258011+0000","last_active":"2024-03-08T08:29:09.995344+0000","last_peered":"2024-03-08T08:29:09.995344+0000","last_clean":"2024-03-08T08:29:09.995344+0000","last_became_active":"2024-03-08T08:29:06.257742+0000","last_became_peered":"2024-03-08T08:29:06.257742+0000","last_unstale":"2024-03-08T08:29:09.995344+0000","last_undegraded":"2024-03-08T08:29:09.995344+0000","last_fullsized":"2024-03-08T08:29:09.995344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:05:29.450798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:09.995182+0000","last_change":"2024-03-08T08:29:06.262040+0000","last_active":"2024-03-08T08:29:09.995182+0000","last_peered":"2024-03-08T08:29:09.995182+0000","last_clean":"2024-03-08T08:29:09.995182+0000","last_became_active":"2024-03-08T08:29:06.258293+0000","last_became_peered":"2024-03-08T08:29:06.258293+0000","last_unstale":"2024-03-08T08:29:09.995182+0000","last_undegraded":"2024-03-08T08:29:09.995182+0000","last_fullsized":"2024-03-08T08:29:09.995182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:36:28.954776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562776+0000","last_change":"2024-03-08T08:29:05.252744+0000","last_active":"2024-03-08T08:29:07.562776+0000","last_peered":"2024-03-08T08:29:07.562776+0000","last_clean":"2024-03-08T08:29:07.562776+0000","last_became_active":"2024-03-08T08:29:05.252365+0000","last_became_peered":"2024-03-08T08:29:05.252365+0000","last_unstale":"2024-03-08T08:29:07.562776+0000","last_undegraded":"2024-03-08T08:29:07.562776+0000","last_fullsized":"2024-03-08T08:29:07.562776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:48:15.439935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.236776+0000","last_change":"2024-03-08T08:29:06.260541+0000","last_active":"2024-03-08T08:29:07.236776+0000","last_peered":"2024-03-08T08:29:07.236776+0000","last_clean":"2024-03-08T08:29:07.236776+0000","last_became_active":"2024-03-08T08:29:06.260283+0000","last_became_peered":"2024-03-08T08:29:06.260283+0000","last_unstale":"2024-03-08T08:29:07.236776+0000","last_undegraded":"2024-03-08T08:29:07.236776+0000","last_fullsized":"2024-03-08T08:29:07.236776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:52:33.000979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.562608+0000","last_change":"2024-03-08T08:29:05.251622+0000","last_active":"2024-03-08T08:29:07.562608+0000","last_peered":"2024-03-08T08:29:07.562608+0000","last_clean":"2024-03-08T08:29:07.562608+0000","last_became_active":"2024-03-08T08:29:05.251346+0000","last_became_peered":"2024-03-08T08:29:05.251346+0000","last_unstale":"2024-03-08T08:29:07.562608+0000","last_undegraded":"2024-03-08T08:29:07.562608+0000","last_fullsized":"2024-03-08T08:29:07.562608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:40:35.023799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905651+0000","last_change":"2024-03-08T08:29:06.245002+0000","last_active":"2024-03-08T08:29:07.905651+0000","last_peered":"2024-03-08T08:29:07.905651+0000","last_clean":"2024-03-08T08:29:07.905651+0000","last_became_active":"2024-03-08T08:29:06.244765+0000","last_became_peered":"2024-03-08T08:29:06.244765+0000","last_unstale":"2024-03-08T08:29:07.905651+0000","last_undegraded":"2024-03-08T08:29:07.905651+0000","last_fullsized":"2024-03-08T08:29:07.905651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:14:06.939392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905631+0000","last_change":"2024-03-08T08:29:05.245332+0000","last_active":"2024-03-08T08:29:07.905631+0000","last_peered":"2024-03-08T08:29:07.905631+0000","last_clean":"2024-03-08T08:29:07.905631+0000","last_became_active":"2024-03-08T08:29:05.245042+0000","last_became_peered":"2024-03-08T08:29:05.245042+0000","last_unstale":"2024-03-08T08:29:07.905631+0000","last_undegraded":"2024-03-08T08:29:07.905631+0000","last_fullsized":"2024-03-08T08:29:07.905631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:36:06.948887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.905465+0000","last_change":"2024-03-08T08:29:06.251546+0000","last_active":"2024-03-08T08:29:07.905465+0000","last_peered":"2024-03-08T08:29:07.905465+0000","last_clean":"2024-03-08T08:29:07.905465+0000","last_became_active":"2024-03-08T08:29:06.251278+0000","last_became_peered":"2024-03-08T08:29:06.251278+0000","last_unstale":"2024-03-08T08:29:07.905465+0000","last_undegraded":"2024-03-08T08:29:07.905465+0000","last_fullsized":"2024-03-08T08:29:07.905465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:05.224783+0000","last_clean_scrub_stamp":"2024-03-08T08:29:05.224783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:29:28.214850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-08T08:29:07.235915+0000","last_change":"2024-03-08T08:29:05.253532+0000","last_active":"2024-03-08T08:29:07.235915+0000","last_peered":"2024-03-08T08:29:07.235915+0000","last_clean":"2024-03-08T08:29:07.235915+0000","last_became_active":"2024-03-08T08:29:05.252904+0000","last_became_peered":"2024-03-08T08:29:05.252904+0000","last_unstale":"2024-03-08T08:29:07.235915+0000","last_undegraded":"2024-03-08T08:29:07.235915+0000","last_fullsized":"2024-03-08T08:29:07.235915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:29:04.219655+0000","last_clean_scrub_stamp":"2024-03-08T08:29:04.219655+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:32:11.309427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":73883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36144,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335696,"statfs":{"total":96636764160,"available":96599752704,"internally_reserved":0,"allocated":815104,"data_stored":664343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":76167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:29:18.451 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-08T08:29:18.452 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-08T08:29:18.452 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-08T08:29:18.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-08T08:29:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.586+0000 7ff355423640 1 Processor -- start 2024-03-08T08:29:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.586+0000 7ff355423640 1 -- start start 2024-03-08T08:29:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff355423640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 0x7ff3500d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff355423640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3500d0f40 0x7ff3500d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff355423640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35015e1a0 con 0x7ff3500d26f0 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff355423640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500b9030 con 0x7ff3500d0f40 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff355423640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500d13a0 con 0x7ff35015ddc0 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff34e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3500d0f40 0x7ff3500d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff34e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3500d0f40 0x7ff3500d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38766/0 (socket says 172.21.15.12:38766) 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff34f7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff3500d26f0 0x7ff3500d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33902/0 (socket says 172.21.15.12:33902) 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff34e7fc640 1 -- 172.21.15.12:0/3048949165 learned_addr learned my addr 172.21.15.12:0/3048949165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.587+0000 7ff34effd640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 0x7ff3500d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34e7fc640 1 -- 172.21.15.12:0/3048949165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 msgr2=0x7ff3500d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34e7fc640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 0x7ff3500d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34e7fc640 1 -- 172.21.15.12:0/3048949165 >> v1:172.21.15.12:6789/0 conn(0x7ff3500d26f0 legacy=0x7ff3500d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34dffb640 1 -- 172.21.15.12:0/3048949165 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461747801 0 0) 0x7ff35015e1a0 con 0x7ff3500d26f0 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34e7fc640 1 -- 172.21.15.12:0/3048949165 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500b8790 con 0x7ff3500d0f40 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34effd640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 0x7ff3500d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34e7fc640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3500d0f40 0x7ff3500d21b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900a35985c62e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34dffb640 1 -- 172.21.15.12:0/3048949165 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338034070 con 0x7ff3500d0f40 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34dffb640 1 -- 172.21.15.12:0/3048949165 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338030840 con 0x7ff3500d0f40 2024-03-08T08:29:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.588+0000 7ff34dffb640 1 -- 172.21.15.12:0/3048949165 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338030c10 con 0x7ff3500d0f40 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 -- 172.21.15.12:0/3048949165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3500d0f40 msgr2=0x7ff3500d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3500d0f40 0x7ff3500d21b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802b980 comp rx=0 tx=0).stop 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 -- 172.21.15.12:0/3048949165 shutdown_connections 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3500d0f40 0x7ff3500d21b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 --2- 172.21.15.12:0/3048949165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff35015ddc0 0x7ff3500d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 -- 172.21.15.12:0/3048949165 >> 172.21.15.12:0/3048949165 conn(0x7ff350072720 msgr2=0x7ff350072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 -- 172.21.15.12:0/3048949165 shutdown_connections 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.589+0000 7ff355423640 1 -- 172.21.15.12:0/3048949165 wait complete. 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.590+0000 7ff355423640 1 Processor -- start 2024-03-08T08:29:18.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.590+0000 7ff355423640 1 -- start start 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3500d0f40 0x7ff35015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3500d26f0 0x7ff3501f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b8d50 con 0x7ff35015ddc0 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500d1d20 con 0x7ff3500d26f0 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff355423640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b9980 con 0x7ff3500d0f40 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38782/0 (socket says 172.21.15.12:38782) 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 -- 172.21.15.12:0/4003862972 learned_addr learned my addr 172.21.15.12:0/4003862972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 -- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3500d0f40 msgr2=0x7ff35015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3500d0f40 0x7ff35015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 -- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3500d26f0 msgr2=0x7ff3501f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3500d26f0 0x7ff3501f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.591+0000 7ff34f7fe640 1 -- 172.21.15.12:0/4003862972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35015c530 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff34f7fe640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c450 tx=0x7ff34800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348018070 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348007500 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3501f7680 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348004a20 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.592+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3501f7c40 con 0x7ff35015ddc0 2024-03-08T08:29:18.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.593+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3501f7e80 con 0x7ff35015ddc0 2024-03-08T08:29:18.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.593+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff34801b070 con 0x7ff35015ddc0 2024-03-08T08:29:18.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.594+0000 7ff3477fe640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.594+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff348089470 con 0x7ff35015ddc0 2024-03-08T08:29:18.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.594+0000 7ff34effd640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.594+0000 7ff34effd640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 0x7ff33006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff3400094c0 tx=0x7ff340009570 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.598+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff3480582f0 con 0x7ff35015ddc0 2024-03-08T08:29:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.816+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3500749c0 con 0x7ff35015ddc0 2024-03-08T08:29:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.816+0000 7ff3477fe640 1 -- 172.21.15.12:0/4003862972 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff348057bf0 con 0x7ff35015ddc0 2024-03-08T08:29:18.817 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:18.818 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-08T08:29:18.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 msgr2=0x7ff33006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 0x7ff33006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff3400094c0 tx=0x7ff340009570 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 msgr2=0x7ff35015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c450 tx=0x7ff34800c910 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 shutdown_connections 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35015ddc0 0x7ff35015b5f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3500d26f0 0x7ff3501f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 --2- 172.21.15.12:0/4003862972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3500d0f40 0x7ff35015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 >> 172.21.15.12:0/4003862972 conn(0x7ff350072720 msgr2=0x7ff3500c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 shutdown_connections 2024-03-08T08:29:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.818+0000 7ff355423640 1 -- 172.21.15.12:0/4003862972 wait complete. 2024-03-08T08:29:18.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-08T08:29:18.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:18.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.930+0000 7f56bca9a640 1 Processor -- start 2024-03-08T08:29:18.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.930+0000 7f56bca9a640 1 -- start start 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56bca9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f56b815ddc0 0x7f56b80d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56bca9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b80d0f40 0x7f56b80d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56bca9a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b815e1a0 con 0x7f56b80d26f0 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56bca9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80b9030 con 0x7f56b80d0f40 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56bca9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80d1d20 con 0x7f56b815ddc0 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56b5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b80d0f40 0x7f56b80d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56b6d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f56b80d26f0 0x7f56b80d2b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33908/0 (socket says 172.21.15.12:33908) 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56b5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b80d0f40 0x7f56b80d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38784/0 (socket says 172.21.15.12:38784) 2024-03-08T08:29:18.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56b6d9d640 1 -- 172.21.15.12:0/790370272 learned_addr learned my addr 172.21.15.12:0/790370272 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.931+0000 7f56b659c640 1 --2- 172.21.15.12:0/790370272 >> v2:172.21.15.156:3300/0 conn(0x7f56b815ddc0 0x7f56b80d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b5d9b640 1 -- 172.21.15.12:0/790370272 >> v2:172.21.15.156:3300/0 conn(0x7f56b815ddc0 msgr2=0x7f56b80d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b5d9b640 1 --2- 172.21.15.12:0/790370272 >> v2:172.21.15.156:3300/0 conn(0x7f56b815ddc0 0x7f56b80d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b5d9b640 1 -- 172.21.15.12:0/790370272 >> v1:172.21.15.12:6789/0 conn(0x7f56b80d26f0 legacy=0x7f56b80d2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b5d9b640 1 -- 172.21.15.12:0/790370272 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80b8790 con 0x7f56b80d0f40 2024-03-08T08:29:18.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b559a640 1 -- 172.21.15.12:0/790370272 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056319107 0 0) 0x7f56b815e1a0 con 0x7f56b80d26f0 2024-03-08T08:29:18.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b5d9b640 1 --2- 172.21.15.12:0/790370272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b80d21b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56b80b66a0 tx=0x7f56ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9528cac57d901222 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b559a640 1 -- 172.21.15.12:0/790370272 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac035020 con 0x7f56b80d0f40 2024-03-08T08:29:18.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b559a640 1 -- 172.21.15.12:0/790370272 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac031830 con 0x7f56b80d0f40 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.932+0000 7f56b559a640 1 -- 172.21.15.12:0/790370272 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac031c00 con 0x7f56b80d0f40 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 -- 172.21.15.12:0/790370272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 msgr2=0x7f56b80d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 --2- 172.21.15.12:0/790370272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b80d21b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56b80b66a0 tx=0x7f56ac02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 -- 172.21.15.12:0/790370272 shutdown_connections 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 --2- 172.21.15.12:0/790370272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b80d21b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 --2- 172.21.15.12:0/790370272 >> v2:172.21.15.156:3300/0 conn(0x7f56b815ddc0 0x7f56b80d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 -- 172.21.15.12:0/790370272 >> 172.21.15.12:0/790370272 conn(0x7f56b8072720 msgr2=0x7f56b8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 -- 172.21.15.12:0/790370272 shutdown_connections 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.933+0000 7f56bca9a640 1 -- 172.21.15.12:0/790370272 wait complete. 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.934+0000 7f56bca9a640 1 Processor -- start 2024-03-08T08:29:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.934+0000 7f56bca9a640 1 -- start start 2024-03-08T08:29:18.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56b80d26f0 0x7f56b8158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56b815ddc0 0x7f56b81f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80b9980 con 0x7f56b80d0f40 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80b8d50 con 0x7f56b815ddc0 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56bca9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80d3250 con 0x7f56b80d26f0 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56b659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56b659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38800/0 (socket says 172.21.15.12:38800) 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.935+0000 7f56b659c640 1 -- 172.21.15.12:0/2587861387 learned_addr learned my addr 172.21.15.12:0/2587861387 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b6d9d640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56b815ddc0 0x7f56b81f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b5d9b640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56b80d26f0 0x7f56b8158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 -- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56b80d26f0 msgr2=0x7f56b8158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56b80d26f0 0x7f56b8158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 -- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56b815ddc0 msgr2=0x7f56b81f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56b815ddc0 0x7f56b81f34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 -- 172.21.15.12:0/2587861387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b81f3ef0 con 0x7f56b80d0f40 2024-03-08T08:29:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56b659c640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d620 tx=0x7f56b000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000b620 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0019070 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b81f41d0 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000ba00 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.936+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8158560 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.937+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b80749c0 con 0x7f56b80d0f40 2024-03-08T08:29:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.937+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56b000bba0 con 0x7f56b80d0f40 2024-03-08T08:29:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.938+0000 7f56a6ffd640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.938+0000 7f56b5d9b640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.938+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56b008a5b0 con 0x7f56b80d0f40 2024-03-08T08:29:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.938+0000 7f56b5d9b640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56ac036000 tx=0x7f56ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:18.942+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f56b00594c0 con 0x7f56b80d0f40 2024-03-08T08:29:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.125+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56b8072f60 con 0x7f56b80d0f40 2024-03-08T08:29:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.126+0000 7f56a6ffd640 1 -- 172.21.15.12:0/2587861387 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f56b0058dc0 con 0x7f56b80d0f40 2024-03-08T08:29:19.128 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:19.128 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 msgr2=0x7f569806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56ac036000 tx=0x7f56ac008030 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 msgr2=0x7f56b8157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d620 tx=0x7f56b000dae0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 shutdown_connections 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56b815ddc0 0x7f56b81f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56b80d26f0 0x7f56b8158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 --2- 172.21.15.12:0/2587861387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b80d0f40 0x7f56b8157080 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 >> 172.21.15.12:0/2587861387 conn(0x7f56b8072720 msgr2=0x7f56b80b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 shutdown_connections 2024-03-08T08:29:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.128+0000 7f56bca9a640 1 -- 172.21.15.12:0/2587861387 wait complete. 2024-03-08T08:29:19.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:19.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:19.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.243+0000 7f8fa5afb640 1 Processor -- start 2024-03-08T08:29:19.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.243+0000 7f8fa5afb640 1 -- start start 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8fa5afb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8fa015cbe0 0x7f8fa00cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8fa5afb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa00cc260 0x7f8fa006abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8fa5afb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00caf60 con 0x7f8fa01561b0 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8fa5afb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00cd620 con 0x7f8fa015cbe0 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8fa5afb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00cb890 con 0x7f8fa00cc260 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8fa015cbe0 0x7f8fa00cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9f7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8fa01561b0 0x7f8fa00cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33910/0 (socket says 172.21.15.12:33910) 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9f7fe640 1 -- 172.21.15.12:0/2237798003 learned_addr learned my addr 172.21.15.12:0/2237798003 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9effd640 1 -- 172.21.15.12:0/2237798003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa00cc260 msgr2=0x7f8fa006abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9effd640 1 --2- 172.21.15.12:0/2237798003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa00cc260 0x7f8fa006abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.244+0000 7f8f9effd640 1 -- 172.21.15.12:0/2237798003 >> v1:172.21.15.12:6789/0 conn(0x7f8fa01561b0 legacy=0x7f8fa00cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.245+0000 7f8f9effd640 1 -- 172.21.15.12:0/2237798003 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00cb240 con 0x7f8fa015cbe0 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.245+0000 7f8f9effd640 1 --2- 172.21.15.12:0/2237798003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa00cbd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8f940097a0 tx=0x7f8f9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a676fa670089ceb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.245+0000 7f8f9e7fc640 1 -- 172.21.15.12:0/2237798003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f940315a0 con 0x7f8fa015cbe0 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.245+0000 7f8f9e7fc640 1 -- 172.21.15.12:0/2237798003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94037070 con 0x7f8fa015cbe0 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.245+0000 7f8f9e7fc640 1 -- 172.21.15.12:0/2237798003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94031960 con 0x7f8fa015cbe0 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 -- 172.21.15.12:0/2237798003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 msgr2=0x7f8fa00cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/2237798003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa00cbd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8f940097a0 tx=0x7f8f9402b840 comp rx=0 tx=0).stop 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 -- 172.21.15.12:0/2237798003 shutdown_connections 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/2237798003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa00cc260 0x7f8fa006abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/2237798003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa00cbd20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 -- 172.21.15.12:0/2237798003 >> 172.21.15.12:0/2237798003 conn(0x7f8fa0099a40 msgr2=0x7f8fa0099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 -- 172.21.15.12:0/2237798003 shutdown_connections 2024-03-08T08:29:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.246+0000 7f8fa5afb640 1 -- 172.21.15.12:0/2237798003 wait complete. 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.247+0000 7f8fa5afb640 1 Processor -- start 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.247+0000 7f8fa5afb640 1 -- start start 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fa00cc260 0x7f8fa00b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa01561b0 0x7f8fa0154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa006b290 con 0x7f8fa015cbe0 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa006b110 con 0x7f8fa00cc260 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8fa5afb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa006b410 con 0x7f8fa01561b0 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38814/0 (socket says 172.21.15.12:38814) 2024-03-08T08:29:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 -- 172.21.15.12:0/3190607568 learned_addr learned my addr 172.21.15.12:0/3190607568 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9f7fe640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fa00cc260 0x7f8fa00b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9effd640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa01561b0 0x7f8fa0154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 -- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa01561b0 msgr2=0x7f8fa0154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa01561b0 0x7f8fa0154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 -- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fa00cc260 msgr2=0x7f8fa00b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fa00cc260 0x7f8fa00b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.248+0000 7f8f9ffff640 1 -- 172.21.15.12:0/3190607568 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00b4130 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8f9ffff640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b970 tx=0x7f8f9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98019070 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800cb60 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa00b2450 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98014440 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.249+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa00b4410 con 0x7f8fa015cbe0 2024-03-08T08:29:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.250+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa00749c0 con 0x7f8fa015cbe0 2024-03-08T08:29:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.250+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f9801c070 con 0x7f8fa015cbe0 2024-03-08T08:29:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.250+0000 7f8f87fff640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 0x7f8f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.251+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f98089f60 con 0x7f8fa015cbe0 2024-03-08T08:29:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.251+0000 7f8f9f7fe640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 0x7f8f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.251+0000 7f8f9f7fe640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 0x7f8f8c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8f90000f60 tx=0x7f8f90009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.255+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8f98058dd0 con 0x7f8fa015cbe0 2024-03-08T08:29:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.436+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fa00d0d60 con 0x7f8fa015cbe0 2024-03-08T08:29:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.437+0000 7f8f87fff640 1 -- 172.21.15.12:0/3190607568 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f980586d0 con 0x7f8fa015cbe0 2024-03-08T08:29:19.438 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:19.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 msgr2=0x7f8f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 0x7f8f8c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8f90000f60 tx=0x7f8f90009040 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 msgr2=0x7f8fa009e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b970 tx=0x7f8f9800be30 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 shutdown_connections 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f8c06b4d0 0x7f8f8c06d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fa015cbe0 0x7f8fa009e0a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fa01561b0 0x7f8fa0154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 --2- 172.21.15.12:0/3190607568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fa00cc260 0x7f8fa00b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 >> 172.21.15.12:0/3190607568 conn(0x7f8fa0099a40 msgr2=0x7f8fa00afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 shutdown_connections 2024-03-08T08:29:19.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.439+0000 7f8fa5afb640 1 -- 172.21.15.12:0/3190607568 wait complete. 2024-03-08T08:29:19.451 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.554+0000 7f6cdd42e640 1 Processor -- start 2024-03-08T08:29:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.554+0000 7f6cdd42e640 1 -- start start 2024-03-08T08:29:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cdd42e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 0x7f6cd80b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cdd42e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cdd42e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd80b7520 con 0x7f6cd80b7080 2024-03-08T08:29:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cdd42e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd80b7800 con 0x7f6cd815dbc0 2024-03-08T08:29:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cdd42e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80b7ae0 con 0x7f6cd80b9760 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd77fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 0x7f6cd80b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6ccffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6cd80b7080 0x7f6cd80d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:39652/0 (socket says 172.21.15.12:39652) 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd77fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:54246/0 (socket says 172.21.15.12:54246) 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6ccffff640 1 -- 172.21.15.12:0/2030048687 learned_addr learned my addr 172.21.15.12:0/2030048687 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd77fe640 1 -- 172.21.15.12:0/2030048687 >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 msgr2=0x7f6cd80b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd77fe640 1 --2- 172.21.15.12:0/2030048687 >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 0x7f6cd80b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.555+0000 7f6cd77fe640 1 -- 172.21.15.12:0/2030048687 >> v1:172.21.15.156:6789/0 conn(0x7f6cd80b7080 legacy=0x7f6cd80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.556+0000 7f6cd77fe640 1 -- 172.21.15.12:0/2030048687 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80b7dc0 con 0x7f6cd80b9760 2024-03-08T08:29:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.556+0000 7f6cd6ffd640 1 --2- 172.21.15.12:0/2030048687 >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 0x7f6cd80b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.556+0000 7f6cd77fe640 1 --2- 172.21.15.12:0/2030048687 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009d90 tx=0x7f6cd002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ef525ac2524ed3e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.556+0000 7f6cd67fc640 1 -- 172.21.15.12:0/2030048687 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0039070 con 0x7f6cd80b9760 2024-03-08T08:29:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cd67fc640 1 -- 172.21.15.12:0/2030048687 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd002cce0 con 0x7f6cd80b9760 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cd67fc640 1 -- 172.21.15.12:0/2030048687 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0034660 con 0x7f6cd80b9760 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 -- 172.21.15.12:0/2030048687 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 msgr2=0x7f6cd80b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/2030048687 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009d90 tx=0x7f6cd002bf30 comp rx=0 tx=0).stop 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 -- 172.21.15.12:0/2030048687 shutdown_connections 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/2030048687 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd80b9760 0x7f6cd80b85c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/2030048687 >> v2:172.21.15.156:3300/0 conn(0x7f6cd815dbc0 0x7f6cd80b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 -- 172.21.15.12:0/2030048687 >> 172.21.15.12:0/2030048687 conn(0x7f6cd8072720 msgr2=0x7f6cd8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 -- 172.21.15.12:0/2030048687 shutdown_connections 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.557+0000 7f6cdd42e640 1 -- 172.21.15.12:0/2030048687 wait complete. 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.558+0000 7f6cdd42e640 1 Processor -- start 2024-03-08T08:29:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.558+0000 7f6cdd42e640 1 -- start start 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cd80b9760 0x7f6cd80caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd815dbc0 0x7f6cd814d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd815dfa0 con 0x7f6cd80b7080 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80d2220 con 0x7f6cd80b9760 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cdd42e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80b8b00 con 0x7f6cd815dbc0 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38828/0 (socket says 172.21.15.12:38828) 2024-03-08T08:29:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 -- 172.21.15.12:0/3652930929 learned_addr learned my addr 172.21.15.12:0/3652930929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6ccffff640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cd80b9760 0x7f6cd80caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd77fe640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd815dbc0 0x7f6cd814d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 -- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd815dbc0 msgr2=0x7f6cd814d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd815dbc0 0x7f6cd814d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 -- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cd80b9760 msgr2=0x7f6cd80caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cd80b9760 0x7f6cd80caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.559+0000 7f6cd6ffd640 1 -- 172.21.15.12:0/3652930929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80cb510 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6cd6ffd640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800e990 tx=0x7f6cc800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8017070 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc800c9e0 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd80cb7f0 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc800cdb0 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.560+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd80cbdb0 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.561+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd80749c0 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.561+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cc801a070 con 0x7f6cd80b7080 2024-03-08T08:29:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.561+0000 7f6ccf7fe640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.562+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cc808a3b0 con 0x7f6cd80b7080 2024-03-08T08:29:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.562+0000 7f6ccffff640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.562+0000 7f6ccffff640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 0x7f6cb806d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8161350 tx=0x7f6cc0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.566+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6cc8059240 con 0x7f6cd80b7080 2024-03-08T08:29:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.744+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6cd80cc190 con 0x7f6cd80b7080 2024-03-08T08:29:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.745+0000 7f6ccf7fe640 1 -- 172.21.15.12:0/3652930929 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6cc8058b40 con 0x7f6cd80b7080 2024-03-08T08:29:19.746 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 msgr2=0x7f6cb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 0x7f6cb806d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8161350 tx=0x7f6cc0009040 comp rx=0 tx=0).stop 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 msgr2=0x7f6cd814ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800e990 tx=0x7f6cc800ee50 comp rx=0 tx=0).stop 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 shutdown_connections 2024-03-08T08:29:19.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cd815dbc0 0x7f6cd814d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cd80b9760 0x7f6cd80caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 --2- 172.21.15.12:0/3652930929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd80b7080 0x7f6cd814ce70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 >> 172.21.15.12:0/3652930929 conn(0x7f6cd8072720 msgr2=0x7f6cd80d24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 shutdown_connections 2024-03-08T08:29:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.747+0000 7f6cdd42e640 1 -- 172.21.15.12:0/3652930929 wait complete. 2024-03-08T08:29:19.759 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.861+0000 7f8440b3a640 1 Processor -- start 2024-03-08T08:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.861+0000 7f8440b3a640 1 -- start start 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8440b3a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f843c06d700 0x7f843c0b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8440b3a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c166190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8440b3a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843c0cdc50 con 0x7f843c0cc360 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8440b3a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843c1666d0 con 0x7f843c06d700 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8440b3a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c0ce540 con 0x7f843c165d20 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f843c06d700 0x7f843c0b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f843c06d700 0x7f843c0b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38844/0 (socket says 172.21.15.12:38844) 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 -- 172.21.15.12:0/1220820568 learned_addr learned my addr 172.21.15.12:0/1220820568 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f843ad9d640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 -- 172.21.15.12:0/1220820568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 msgr2=0x7f843c166190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c166190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 -- 172.21.15.12:0/1220820568 >> v1:172.21.15.156:6789/0 conn(0x7f843c0cc360 legacy=0x7f843c0b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.862+0000 7f8439d9b640 1 -- 172.21.15.12:0/1220820568 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0b3a80 con 0x7f843c06d700 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.863+0000 7f8439d9b640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c0b34a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e816f5ea567d93d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.863+0000 7f843959a640 1 -- 172.21.15.12:0/1220820568 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424035020 con 0x7f843c06d700 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.863+0000 7f843959a640 1 -- 172.21.15.12:0/1220820568 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424031640 con 0x7f843c06d700 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.863+0000 7f843959a640 1 -- 172.21.15.12:0/1220820568 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842402d040 con 0x7f843c06d700 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 -- 172.21.15.12:0/1220820568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 msgr2=0x7f843c0b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c0b34a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 -- 172.21.15.12:0/1220820568 shutdown_connections 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c166190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 --2- 172.21.15.12:0/1220820568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c0b34a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 -- 172.21.15.12:0/1220820568 >> 172.21.15.12:0/1220820568 conn(0x7f843c06abd0 msgr2=0x7f843c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 -- 172.21.15.12:0/1220820568 shutdown_connections 2024-03-08T08:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.864+0000 7f8440b3a640 1 -- 172.21.15.12:0/1220820568 wait complete. 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.865+0000 7f8440b3a640 1 Processor -- start 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.865+0000 7f8440b3a640 1 -- start start 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f843c0cc360 0x7f843c1fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c0cc740 con 0x7f843c06d700 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c0ce1d0 con 0x7f843c0cc360 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8440b3a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c162750 con 0x7f843c165d20 2024-03-08T08:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38848/0 (socket says 172.21.15.12:38848) 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 -- 172.21.15.12:0/723114056 learned_addr learned my addr 172.21.15.12:0/723114056 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f8439d9b640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f843c0cc360 0x7f843c1fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843ad9d640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 -- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 msgr2=0x7f843c148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 -- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f843c0cc360 msgr2=0x7f843c1fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f843c0cc360 0x7f843c1fb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 -- 172.21.15.12:0/723114056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c1fbc60 con 0x7f843c06d700 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843ad9d640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.866+0000 7f843a59c640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f843400c430 tx=0x7f843400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.867+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434011790 con 0x7f843c06d700 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.867+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8434017070 con 0x7f843c06d700 2024-03-08T08:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.867+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c1fbf30 con 0x7f843c06d700 2024-03-08T08:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.867+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434011bb0 con 0x7f843c06d700 2024-03-08T08:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.867+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843c1fc4f0 con 0x7f843c06d700 2024-03-08T08:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.868+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c0749c0 con 0x7f843c06d700 2024-03-08T08:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.868+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f843401c460 con 0x7f843c06d700 2024-03-08T08:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.868+0000 7f8432ffd640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.869+0000 7f8439d9b640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.869+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f843408a070 con 0x7f843c06d700 2024-03-08T08:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.869+0000 7f8439d9b640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f842403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:19.873+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8434058ef0 con 0x7f843c06d700 2024-03-08T08:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.054+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f843c0b1450 con 0x7f843c06d700 2024-03-08T08:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.054+0000 7f8432ffd640 1 -- 172.21.15.12:0/723114056 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84340587f0 con 0x7f843c06d700 2024-03-08T08:29:20.055 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 msgr2=0x7f841c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f842403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 msgr2=0x7f843c165740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f843400c430 tx=0x7f843400c8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 shutdown_connections 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f843c165d20 0x7f843c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f843c0cc360 0x7f843c1fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 --2- 172.21.15.12:0/723114056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f843c06d700 0x7f843c165740 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.056+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 >> 172.21.15.12:0/723114056 conn(0x7f843c06abd0 msgr2=0x7f843c160350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.057+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 shutdown_connections 2024-03-08T08:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.057+0000 7f8440b3a640 1 -- 172.21.15.12:0/723114056 wait complete. 2024-03-08T08:29:20.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.168+0000 7f679bd5c640 1 Processor -- start 2024-03-08T08:29:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.168+0000 7f679bd5c640 1 -- start start 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679bd5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679bd5c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f67940d31c0 0x7f67940b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679bd5c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679415e1a0 con 0x7f679415ddc0 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679bd5c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67940b9030 con 0x7f67940d31c0 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679bd5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940d1d20 con 0x7f67940d1110 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f679a2f9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f67940d31c0 0x7f67940b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:54290/0 (socket says 172.21.15.12:54290) 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 -- 172.21.15.12:0/978392964 learned_addr learned my addr 172.21.15.12:0/978392964 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 -- 172.21.15.12:0/978392964 >> v2:172.21.15.156:3300/0 conn(0x7f67940d31c0 msgr2=0x7f67940b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 --2- 172.21.15.12:0/978392964 >> v2:172.21.15.156:3300/0 conn(0x7f67940d31c0 0x7f67940b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 -- 172.21.15.12:0/978392964 >> v1:172.21.15.156:6789/0 conn(0x7f679415ddc0 legacy=0x7f67940d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f67992f7640 1 -- 172.21.15.12:0/978392964 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940b8790 con 0x7f67940d1110 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.169+0000 7f6798af6640 1 -- 172.21.15.12:0/978392964 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893643251 0 0) 0x7f679415e1a0 con 0x7f679415ddc0 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.170+0000 7f67992f7640 1 --2- 172.21.15.12:0/978392964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6784009870 tx=0x7f678402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4064f566c6c3abc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.170+0000 7f6798af6640 1 -- 172.21.15.12:0/978392964 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784034070 con 0x7f67940d1110 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.170+0000 7f6798af6640 1 -- 172.21.15.12:0/978392964 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784030840 con 0x7f67940d1110 2024-03-08T08:29:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.170+0000 7f6798af6640 1 -- 172.21.15.12:0/978392964 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784030c10 con 0x7f67940d1110 2024-03-08T08:29:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 -- 172.21.15.12:0/978392964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 msgr2=0x7f67940b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 --2- 172.21.15.12:0/978392964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6784009870 tx=0x7f678402b980 comp rx=0 tx=0).stop 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 -- 172.21.15.12:0/978392964 shutdown_connections 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 --2- 172.21.15.12:0/978392964 >> v2:172.21.15.156:3300/0 conn(0x7f67940d31c0 0x7f67940b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 --2- 172.21.15.12:0/978392964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f67940b6be0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 -- 172.21.15.12:0/978392964 >> 172.21.15.12:0/978392964 conn(0x7f6794072720 msgr2=0x7f6794072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 -- 172.21.15.12:0/978392964 shutdown_connections 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.171+0000 7f679bd5c640 1 -- 172.21.15.12:0/978392964 wait complete. 2024-03-08T08:29:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.172+0000 7f679bd5c640 1 Processor -- start 2024-03-08T08:29:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.172+0000 7f679bd5c640 1 -- start start 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f679415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f679415ddc0 0x7f67941f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67940b9980 con 0x7f67940d31c0 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67940b8d50 con 0x7f679415ddc0 2024-03-08T08:29:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940b7660 con 0x7f67940d1110 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38864/0 (socket says 172.21.15.12:38864) 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 -- 172.21.15.12:0/2760295825 learned_addr learned my addr 172.21.15.12:0/2760295825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f6799af8640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f679415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679a2f9640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f679415ddc0 0x7f67941f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 -- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 msgr2=0x7f679415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f679415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 -- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f679415ddc0 msgr2=0x7f67941f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f679415ddc0 0x7f67941f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 -- 172.21.15.12:0/2760295825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679415e530 con 0x7f67940d31c0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f67992f7640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c00 tx=0x7f67840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784034040 con 0x7f67940d31c0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784030640 con 0x7f67940d31c0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679415e7d0 con 0x7f67940d31c0 2024-03-08T08:29:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784039d40 con 0x7f67940d31c0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.173+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679415ed90 con 0x7f67940d31c0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.175+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940749c0 con 0x7f67940d31c0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.175+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f678404b060 con 0x7f67940d31c0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.175+0000 7f678a7fc640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.175+0000 7f6799af8640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.175+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67840abfe0 con 0x7f67940d31c0 2024-03-08T08:29:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.176+0000 7f6799af8640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 0x7f677806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6790007fd0 tx=0x7f6790009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.180+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f678407af00 con 0x7f67940d31c0 2024-03-08T08:29:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.399+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6794160180 con 0x7f67940d31c0 2024-03-08T08:29:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.399+0000 7f678a7fc640 1 -- 172.21.15.12:0/2760295825 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f678407a800 con 0x7f67940d31c0 2024-03-08T08:29:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:20.401 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.401+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 msgr2=0x7f677806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.401+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 0x7f677806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6790007fd0 tx=0x7f6790009040 comp rx=0 tx=0).stop 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.401+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 msgr2=0x7f6794160db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.401+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c00 tx=0x7f67840046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.401+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 shutdown_connections 2024-03-08T08:29:20.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f679415ddc0 0x7f67941f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67940d31c0 0x7f6794160db0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 --2- 172.21.15.12:0/2760295825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67940d1110 0x7f679415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 >> 172.21.15.12:0/2760295825 conn(0x7f6794072720 msgr2=0x7f67940d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 shutdown_connections 2024-03-08T08:29:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.402+0000 7f679bd5c640 1 -- 172.21.15.12:0/2760295825 wait complete. 2024-03-08T08:29:20.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.514+0000 7f990e2eb640 1 Processor -- start 2024-03-08T08:29:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.514+0000 7f990e2eb640 1 -- start start 2024-03-08T08:29:20.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990e2eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f99080cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990e2eb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f990809a310 0x7f990806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990e2eb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080d0110 con 0x7f99080d11b0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990e2eb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990815f580 con 0x7f990809a310 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990e2eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080d03f0 con 0x7f990815f1a0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f99077fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f99080d11b0 0x7f9908099c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33918/0 (socket says 172.21.15.12:33918) 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f99077fe640 1 -- 172.21.15.12:0/358406958 learned_addr learned my addr 172.21.15.12:0/358406958 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f990c888640 1 --2- 172.21.15.12:0/358406958 >> v2:172.21.15.156:3300/0 conn(0x7f990809a310 0x7f990806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816076354 0 0) 0x7f99080d0110 con 0x7f99080d11b0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f0003520 con 0x7f99080d11b0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9907fff640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f99080cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4117959154 0 0) 0x7f98f0003520 con 0x7f99080d11b0 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 msgr2=0x7f99080cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f99080cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 >> v2:172.21.15.156:3300/0 conn(0x7f990809a310 msgr2=0x7f990806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 --2- 172.21.15.12:0/358406958 >> v2:172.21.15.156:3300/0 conn(0x7f990809a310 0x7f990806cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080d06d0 con 0x7f99080d11b0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f98f8001670 con 0x7f99080d11b0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 >> v1:172.21.15.12:6789/0 conn(0x7f99080d11b0 legacy=0x7f9908099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f00039b0 0x7f98f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98f0004800 0x7f98f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f98f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080d03f0 con 0x7f98f0004800 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f000b9c0 con 0x7f98f00039b0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.515+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f000bb40 con 0x7f990815f1a0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98f0004800 0x7f98f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f990c888640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f98f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f9907fff640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f00039b0 0x7f98f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 -- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 msgr2=0x7f98f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f990815f1a0 0x7f98f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 -- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f00039b0 msgr2=0x7f98f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f00039b0 0x7f98f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 -- 172.21.15.12:0/358406958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f0003520 con 0x7f98f0004800 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.516+0000 7f99077fe640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98f0004800 0x7f98f0004c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f98f8001050 tx=0x7f98f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0be1bdddf3a7140 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8007d70 con 0x7f98f0004800 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f8030070 con 0x7f98f0004800 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f9906ffd640 1 -- 172.21.15.12:0/358406958 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f802cd80 con 0x7f98f0004800 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 -- 172.21.15.12:0/358406958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98f0004800 msgr2=0x7f98f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 --2- 172.21.15.12:0/358406958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98f0004800 0x7f98f0004c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f98f8001050 tx=0x7f98f0005830 comp rx=0 tx=0).stop 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990c888640 1 -- 172.21.15.12:0/358406958 reap_dead start 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 -- 172.21.15.12:0/358406958 shutdown_connections 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 -- 172.21.15.12:0/358406958 >> 172.21.15.12:0/358406958 conn(0x7f99080b3490 msgr2=0x7f99080b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 -- 172.21.15.12:0/358406958 shutdown_connections 2024-03-08T08:29:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.517+0000 7f990e2eb640 1 -- 172.21.15.12:0/358406958 wait complete. 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.518+0000 7f990e2eb640 1 Processor -- start 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.518+0000 7f990e2eb640 1 -- start start 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f0004800 0x7f99080c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99080cb820 0x7f9908156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990806bf00 con 0x7f990809d9f0 2024-03-08T08:29:20.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080d0c90 con 0x7f98f0004800 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990e2eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080cfb10 con 0x7f99080cb820 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38888/0 (socket says 172.21.15.12:38888) 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 -- 172.21.15.12:0/3002305048 learned_addr learned my addr 172.21.15.12:0/3002305048 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f99077fe640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99080cb820 0x7f9908156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 -- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99080cb820 msgr2=0x7f9908156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99080cb820 0x7f9908156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f990c888640 1 -- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f0004800 msgr2=0x7f99080c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.519+0000 7f9907fff640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f0004800 0x7f99080c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f990c888640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f0004800 0x7f99080c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f990c888640 1 -- 172.21.15.12:0/3002305048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080cbca0 con 0x7f990809d9f0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f990c888640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99080cbef0 tx=0x7f990000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900019070 con 0x7f990809d9f0 2024-03-08T08:29:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900003990 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99080c9b60 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900003d80 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.520+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99080ca120 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.521+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99080749c0 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.521+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f990001c070 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.522+0000 7f9904ff9640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 0x7f98f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.522+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f990008aba0 con 0x7f990809d9f0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.522+0000 7f9907fff640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 0x7f98f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.522+0000 7f9907fff640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 0x7f98f406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098a0 tx=0x7f98fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.526+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9900010040 con 0x7f990809d9f0 2024-03-08T08:29:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.706+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f990806bce0 con 0x7f990809d9f0 2024-03-08T08:29:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.708+0000 7f9904ff9640 1 -- 172.21.15.12:0/3002305048 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f9900059ab0 con 0x7f990809d9f0 2024-03-08T08:29:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:20.709 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 msgr2=0x7f98f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 0x7f98f406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098a0 tx=0x7f98fc008040 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 msgr2=0x7f990809de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99080cbef0 tx=0x7f990000f580 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 shutdown_connections 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98f406b4d0 0x7f98f406d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990809d9f0 0x7f990809de70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99080cb820 0x7f9908156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 --2- 172.21.15.12:0/3002305048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98f0004800 0x7f99080c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.709+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 >> 172.21.15.12:0/3002305048 conn(0x7f99080b3490 msgr2=0x7f99080b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.710+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 shutdown_connections 2024-03-08T08:29:20.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.710+0000 7f990e2eb640 1 -- 172.21.15.12:0/3002305048 wait complete. 2024-03-08T08:29:20.721 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:20.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.824+0000 7f3a757f4640 1 Processor -- start 2024-03-08T08:29:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 -- start start 2024-03-08T08:29:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a700d0250 0x7f3a700d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 0x7f3a700d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700b6be0 con 0x7f3a7015dec0 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700d2140 con 0x7f3a700d0250 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a757f4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7015e2a0 con 0x7f3a700d25d0 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a6e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a700d0250 0x7f3a700d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a6e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a700d0250 0x7f3a700d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38896/0 (socket says 172.21.15.12:38896) 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.825+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/688856031 learned_addr learned my addr 172.21.15.12:0/688856031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6f7fe640 1 --2- 172.21.15.12:0/688856031 >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 0x7f3a700d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/688856031 >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 msgr2=0x7f3a700d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6e7fc640 1 --2- 172.21.15.12:0/688856031 >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 0x7f3a700d2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/688856031 >> v1:172.21.15.156:6789/0 conn(0x7f3a7015dec0 legacy=0x7f3a70161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/688856031 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a701620a0 con 0x7f3a700d0250 2024-03-08T08:29:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6f7fe640 1 --2- 172.21.15.12:0/688856031 >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 0x7f3a700d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.826+0000 7f3a6e7fc640 1 --2- 172.21.15.12:0/688856031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d0250 0x7f3a700d06b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a7006bba0 tx=0x7f3a5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41902f7dbc2899d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a6dffb640 1 -- 172.21.15.12:0/688856031 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58035020 con 0x7f3a700d0250 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a6dffb640 1 -- 172.21.15.12:0/688856031 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58031870 con 0x7f3a700d0250 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a6dffb640 1 -- 172.21.15.12:0/688856031 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58031c40 con 0x7f3a700d0250 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 -- 172.21.15.12:0/688856031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d0250 msgr2=0x7f3a700d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 --2- 172.21.15.12:0/688856031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d0250 0x7f3a700d06b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a7006bba0 tx=0x7f3a5802b910 comp rx=0 tx=0).stop 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 -- 172.21.15.12:0/688856031 shutdown_connections 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 --2- 172.21.15.12:0/688856031 >> v2:172.21.15.156:3300/0 conn(0x7f3a700d25d0 0x7f3a700d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 --2- 172.21.15.12:0/688856031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d0250 0x7f3a700d06b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 -- 172.21.15.12:0/688856031 >> 172.21.15.12:0/688856031 conn(0x7f3a7006c240 msgr2=0x7f3a7006c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 -- 172.21.15.12:0/688856031 shutdown_connections 2024-03-08T08:29:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.827+0000 7f3a757f4640 1 -- 172.21.15.12:0/688856031 wait complete. 2024-03-08T08:29:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.828+0000 7f3a757f4640 1 Processor -- start 2024-03-08T08:29:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 -- start start 2024-03-08T08:29:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 0x7f3a700ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a7015dec0 0x7f3a7015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70162630 con 0x7f3a700d25d0 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70162350 con 0x7f3a700d0250 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a757f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700d1dd0 con 0x7f3a7015dec0 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a6e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a6e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38908/0 (socket says 172.21.15.12:38908) 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/1061160751 learned_addr learned my addr 172.21.15.12:0/1061160751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.829+0000 7f3a6effd640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 0x7f3a700ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6f7fe640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a7015dec0 0x7f3a7015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a7015dec0 msgr2=0x7f3a7015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a7015dec0 0x7f3a7015e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 msgr2=0x7f3a700ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 0x7f3a700ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 -- 172.21.15.12:0/1061160751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700cbd70 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6effd640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 0x7f3a700ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a6e7fc640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58040070 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5803a420 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70160dd0 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58035040 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.830+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a700cc010 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.831+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a701612f0 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.831+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a58038020 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.832+0000 7f3a677fe640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.832+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a580ac180 con 0x7f3a700d25d0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.832+0000 7f3a6effd640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.832+0000 7f3a6effd640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 0x7f3a5006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009fc0 tx=0x7f3a60009550 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:20.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:20.836+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a701612f0 con 0x7f3a700d25d0 2024-03-08T08:29:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.014+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a700749c0 con 0x7f3a700d25d0 2024-03-08T08:29:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.014+0000 7f3a677fe640 1 -- 172.21.15.12:0/1061160751 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a700b6be0 con 0x7f3a700d25d0 2024-03-08T08:29:21.016 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 msgr2=0x7f3a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 0x7f3a5006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009fc0 tx=0x7f3a60009550 comp rx=0 tx=0).stop 2024-03-08T08:29:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 msgr2=0x7f3a701606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a580046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 shutdown_connections 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a7015dec0 0x7f3a7015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a700d25d0 0x7f3a701606a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 --2- 172.21.15.12:0/1061160751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a700d0250 0x7f3a700ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 >> 172.21.15.12:0/1061160751 conn(0x7f3a7006c240 msgr2=0x7f3a700b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 shutdown_connections 2024-03-08T08:29:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.017+0000 7f3a757f4640 1 -- 172.21.15.12:0/1061160751 wait complete. 2024-03-08T08:29:21.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.131+0000 7fed8a884640 1 Processor -- start 2024-03-08T08:29:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.131+0000 7fed8a884640 1 -- start start 2024-03-08T08:29:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed8a884640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed8406fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed8a884640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fed840b2c60 0x7fed840cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed8a884640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed84161b30 con 0x7fed8406b450 2024-03-08T08:29:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed8a884640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed84070650 con 0x7fed840b2c60 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed8a884640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed84070090 con 0x7fed8406b070 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed837fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fed8406b450 0x7fed841622f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33932/0 (socket says 172.21.15.12:33932) 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.132+0000 7fed837fe640 1 -- 172.21.15.12:0/192702039 learned_addr learned my addr 172.21.15.12:0/192702039 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed83fff640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed8406fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed88e21640 1 --2- 172.21.15.12:0/192702039 >> v2:172.21.15.156:3300/0 conn(0x7fed840b2c60 0x7fed840cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153641393 0 0) 0x7fed84161b30 con 0x7fed8406b450 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70003520 con 0x7fed8406b450 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58027852 0 0) 0x7fed70003520 con 0x7fed8406b450 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 msgr2=0x7fed8406fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed8406fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 >> v2:172.21.15.156:3300/0 conn(0x7fed840b2c60 msgr2=0x7fed840cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 --2- 172.21.15.12:0/192702039 >> v2:172.21.15.156:3300/0 conn(0x7fed840b2c60 0x7fed840cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84070370 con 0x7fed8406b450 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fed6c001670 con 0x7fed8406b450 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 >> v1:172.21.15.12:6789/0 conn(0x7fed8406b450 legacy=0x7fed841622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84070650 con 0x7fed70004800 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84070090 con 0x7fed700039b0 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000b9c0 con 0x7fed8406b070 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed83fff640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed88e21640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 -- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 msgr2=0x7fed700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed8406b070 0x7fed700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 -- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed700039b0 msgr2=0x7fed70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.133+0000 7fed837fe640 1 -- 172.21.15.12:0/192702039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70003520 con 0x7fed70004800 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.134+0000 7fed837fe640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c210 tx=0x7fed70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90e5c49b7640af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.134+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c007d30 con 0x7fed70004800 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.134+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c030070 con 0x7fed70004800 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.134+0000 7fed82ffd640 1 -- 172.21.15.12:0/192702039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c02cd60 con 0x7fed70004800 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 -- 172.21.15.12:0/192702039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 msgr2=0x7fed70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 --2- 172.21.15.12:0/192702039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c210 tx=0x7fed70005830 comp rx=0 tx=0).stop 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed88e21640 1 -- 172.21.15.12:0/192702039 reap_dead start 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 -- 172.21.15.12:0/192702039 shutdown_connections 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 -- 172.21.15.12:0/192702039 >> 172.21.15.12:0/192702039 conn(0x7fed84099a40 msgr2=0x7fed840b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 -- 172.21.15.12:0/192702039 shutdown_connections 2024-03-08T08:29:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.135+0000 7fed8a884640 1 -- 172.21.15.12:0/192702039 wait complete. 2024-03-08T08:29:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.136+0000 7fed8a884640 1 Processor -- start 2024-03-08T08:29:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.136+0000 7fed8a884640 1 -- start start 2024-03-08T08:29:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 0x7fed840cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed84155a90 0x7fed840c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8406fab0 con 0x7fed840ca510 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed840b3110 con 0x7fed70004800 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8406fc30 con 0x7fed84155a90 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38936/0 (socket says 172.21.15.12:38936) 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 -- 172.21.15.12:0/2001787676 learned_addr learned my addr 172.21.15.12:0/2001787676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed83fff640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 0x7fed840cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed837fe640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed84155a90 0x7fed840c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 -- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed84155a90 msgr2=0x7fed840c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed84155a90 0x7fed840c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 -- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 msgr2=0x7fed840cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 0x7fed840cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 -- 172.21.15.12:0/2001787676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed840cc220 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed83fff640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 0x7fed840cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed88e21640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed840cf930 tx=0x7fed7c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00fce0 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c00fe80 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed840cc500 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c015670 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.137+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed840ccac0 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.138+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed840749c0 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.139+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed7c015810 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.139+0000 7fed80ff9640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 0x7fed6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.139+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed7c08afc0 con 0x7fed840ca510 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.139+0000 7fed83fff640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 0x7fed6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.140+0000 7fed83fff640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 0x7fed6406d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fed84155960 tx=0x7fed74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.144+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fed7c019030 con 0x7fed840ca510 2024-03-08T08:29:21.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.326+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed84099e50 con 0x7fed840ca510 2024-03-08T08:29:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.327+0000 7fed80ff9640 1 -- 172.21.15.12:0/2001787676 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed7c010040 con 0x7fed840ca510 2024-03-08T08:29:21.328 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 msgr2=0x7fed6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 0x7fed6406d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fed84155960 tx=0x7fed74008040 comp rx=0 tx=0).stop 2024-03-08T08:29:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 msgr2=0x7fed840c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed840cf930 tx=0x7fed7c00f1b0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 shutdown_connections 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fed6406b4d0 0x7fed6406d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed840ca510 0x7fed840c7890 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fed84155a90 0x7fed840c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 --2- 172.21.15.12:0/2001787676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed70004800 0x7fed840cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 >> 172.21.15.12:0/2001787676 conn(0x7fed84099a40 msgr2=0x7fed840d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 shutdown_connections 2024-03-08T08:29:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.329+0000 7fed8a884640 1 -- 172.21.15.12:0/2001787676 wait complete. 2024-03-08T08:29:21.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:21.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.444+0000 7f1166864640 1 Processor -- start 2024-03-08T08:29:21.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.444+0000 7f1166864640 1 -- start start 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1166864640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1166864640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 0x7f11600b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1166864640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116015e1a0 con 0x7f116015ddc0 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1166864640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11600b9030 con 0x7f11600d1110 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1166864640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11600d1d20 con 0x7f11600d31c0 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115f7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f116015ddc0 0x7f11600d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:39680/0 (socket says 172.21.15.12:39680) 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1164e01640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 0x7f11600b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115ffff640 1 -- 172.21.15.12:0/1204924518 learned_addr learned my addr 172.21.15.12:0/1204924518 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115f7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:48186/0 (socket says 172.21.15.12:48186) 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1164e01640 1 -- 172.21.15.12:0/1204924518 >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 msgr2=0x7f11600b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1164e01640 1 --2- 172.21.15.12:0/1204924518 >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1164e01640 1 -- 172.21.15.12:0/1204924518 >> v1:172.21.15.156:6789/0 conn(0x7f116015ddc0 legacy=0x7f11600d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f1164e01640 1 -- 172.21.15.12:0/1204924518 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11600b8790 con 0x7f11600d31c0 2024-03-08T08:29:21.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115f7fe640 1 --2- 172.21.15.12:0/1204924518 >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:21.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.445+0000 7f115effd640 1 -- 172.21.15.12:0/1204924518 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823925273 0 0) 0x7f116015e1a0 con 0x7f116015ddc0 2024-03-08T08:29:21.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.446+0000 7f1164e01640 1 --2- 172.21.15.12:0/1204924518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 0x7f11600b7120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1158009d90 tx=0x7f115802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea1d1f6713808b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.446+0000 7f115effd640 1 -- 172.21.15.12:0/1204924518 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158039070 con 0x7f11600d31c0 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.446+0000 7f115effd640 1 -- 172.21.15.12:0/1204924518 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115802cc50 con 0x7f11600d31c0 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.446+0000 7f115effd640 1 -- 172.21.15.12:0/1204924518 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158034440 con 0x7f11600d31c0 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 -- 172.21.15.12:0/1204924518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 msgr2=0x7f11600b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 --2- 172.21.15.12:0/1204924518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 0x7f11600b7120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1158009d90 tx=0x7f115802bea0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 -- 172.21.15.12:0/1204924518 shutdown_connections 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 --2- 172.21.15.12:0/1204924518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d31c0 0x7f11600b7120 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 --2- 172.21.15.12:0/1204924518 >> v2:172.21.15.156:3300/0 conn(0x7f11600d1110 0x7f11600b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 -- 172.21.15.12:0/1204924518 >> 172.21.15.12:0/1204924518 conn(0x7f1160072720 msgr2=0x7f1160072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 -- 172.21.15.12:0/1204924518 shutdown_connections 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.447+0000 7f1166864640 1 -- 172.21.15.12:0/1204924518 wait complete. 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.448+0000 7f1166864640 1 Processor -- start 2024-03-08T08:29:21.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.448+0000 7f1166864640 1 -- start start 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d1110 0x7f116014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11600d31c0 0x7f1160160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11600b9980 con 0x7f116015ddc0 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11600b8d50 con 0x7f11600d31c0 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1166864640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11600b7660 con 0x7f11600d1110 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38940/0 (socket says 172.21.15.12:38940) 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 -- 172.21.15.12:0/189047052 learned_addr learned my addr 172.21.15.12:0/189047052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f115f7fe640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11600d31c0 0x7f1160160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f115ffff640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d1110 0x7f116014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 -- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d1110 msgr2=0x7f116014dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d1110 0x7f116014dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 -- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11600d31c0 msgr2=0x7f1160160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11600d31c0 0x7f1160160390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.449+0000 7f1164e01640 1 -- 172.21.15.12:0/189047052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116014e290 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f1164e01640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1158009eb0 tx=0x7f1158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158039050 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158045070 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1160160bc0 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158009450 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.450+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1160161180 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.451+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11600749c0 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.451+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f115803e610 con 0x7f116015ddc0 2024-03-08T08:29:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.451+0000 7f115cff9640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 0x7f114c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.452+0000 7f115ffff640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 0x7f114c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.452+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11580aca80 con 0x7f116015ddc0 2024-03-08T08:29:21.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.452+0000 7f115ffff640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 0x7f114c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1154007fd0 tx=0x7f1154007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.456+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f115807b340 con 0x7f116015ddc0 2024-03-08T08:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.635+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f11600d0ce0 con 0x7f116015ddc0 2024-03-08T08:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.635+0000 7f115cff9640 1 -- 172.21.15.12:0/189047052 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f115807f3e0 con 0x7f116015ddc0 2024-03-08T08:29:21.637 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 msgr2=0x7f114c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 0x7f114c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1154007fd0 tx=0x7f1154007de0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 msgr2=0x7f11601e9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1158009eb0 tx=0x7f1158004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 shutdown_connections 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f114c06b4d0 0x7f114c06d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f116015ddc0 0x7f11601e9040 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11600d31c0 0x7f1160160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 --2- 172.21.15.12:0/189047052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11600d1110 0x7f116014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 >> 172.21.15.12:0/189047052 conn(0x7f1160072720 msgr2=0x7f11600b7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 shutdown_connections 2024-03-08T08:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.637+0000 7f1166864640 1 -- 172.21.15.12:0/189047052 wait complete. 2024-03-08T08:29:21.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.749+0000 7f87d87b6640 1 Processor -- start 2024-03-08T08:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.749+0000 7f87d87b6640 1 -- start start 2024-03-08T08:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f87d006c240 0x7f87d006c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 0x7f87d015e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87d00b5550 0x7f87d00b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d00b6b30 con 0x7f87d00b5550 2024-03-08T08:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d006cb60 con 0x7f87d006c240 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d87b6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d015ebc0 con 0x7f87d00b9420 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87d00b5550 0x7f87d00b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87d00b5550 0x7f87d00b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38954/0 (socket says 172.21.15.12:38954) 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 -- 172.21.15.12:0/1041260265 learned_addr learned my addr 172.21.15.12:0/1041260265 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d5d51640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 0x7f87d015e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6552640 1 --2- 172.21.15.12:0/1041260265 >> v2:172.21.15.156:3300/0 conn(0x7f87d006c240 0x7f87d006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 -- 172.21.15.12:0/1041260265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 msgr2=0x7f87d015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 0x7f87d015e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 -- 172.21.15.12:0/1041260265 >> v2:172.21.15.156:3300/0 conn(0x7f87d006c240 msgr2=0x7f87d006c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 --2- 172.21.15.12:0/1041260265 >> v2:172.21.15.156:3300/0 conn(0x7f87d006c240 0x7f87d006c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d6d53640 1 -- 172.21.15.12:0/1041260265 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d015eea0 con 0x7f87d00b5550 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.750+0000 7f87d5d51640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 0x7f87d015e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.751+0000 7f87d6d53640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d00b5550 0x7f87d00b5a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f87d006cf70 tx=0x7f87c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82b60814eefa892 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.751+0000 7f87d5550640 1 -- 172.21.15.12:0/1041260265 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0034070 con 0x7f87d00b5550 2024-03-08T08:29:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.751+0000 7f87d5550640 1 -- 172.21.15.12:0/1041260265 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c0030860 con 0x7f87d00b5550 2024-03-08T08:29:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.751+0000 7f87d5550640 1 -- 172.21.15.12:0/1041260265 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0030c30 con 0x7f87d00b5550 2024-03-08T08:29:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 -- 172.21.15.12:0/1041260265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d00b5550 msgr2=0x7f87d00b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d00b5550 0x7f87d00b5a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f87d006cf70 tx=0x7f87c002b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 -- 172.21.15.12:0/1041260265 shutdown_connections 2024-03-08T08:29:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d00b5550 0x7f87d00b5a00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1041260265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b9420 0x7f87d015e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1041260265 >> v2:172.21.15.156:3300/0 conn(0x7f87d006c240 0x7f87d006c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 -- 172.21.15.12:0/1041260265 >> 172.21.15.12:0/1041260265 conn(0x7f87d0072900 msgr2=0x7f87d00d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 -- 172.21.15.12:0/1041260265 shutdown_connections 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.752+0000 7f87d87b6640 1 -- 172.21.15.12:0/1041260265 wait complete. 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.753+0000 7f87d87b6640 1 Processor -- start 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.753+0000 7f87d87b6640 1 -- start start 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b5550 0x7f87d01f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87d00b9420 0x7f87d00b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d015bc30 con 0x7f87d006c240 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00d1f90 con 0x7f87d00b9420 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d015f180 con 0x7f87d00b5550 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38966/0 (socket says 172.21.15.12:38966) 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 -- 172.21.15.12:0/1005360488 learned_addr learned my addr 172.21.15.12:0/1005360488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 -- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b5550 msgr2=0x7f87d01f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b5550 0x7f87d01f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 -- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87d00b9420 msgr2=0x7f87d00b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87d00b9420 0x7f87d00b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 -- 172.21.15.12:0/1005360488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d00cb360 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d6552640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f87c8007e40 tx=0x7f87c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8014620 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8019070 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d00cb640 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8014a00 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.754+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d00cbc00 con 0x7f87d006c240 2024-03-08T08:29:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.755+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d00749c0 con 0x7f87d006c240 2024-03-08T08:29:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.756+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87c8014ba0 con 0x7f87d006c240 2024-03-08T08:29:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.756+0000 7f87ceffd640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.756+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87c808ab60 con 0x7f87d006c240 2024-03-08T08:29:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.756+0000 7f87d5d51640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.757+0000 7f87d5d51640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 0x7f87bc06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f87d01559e0 tx=0x7f87c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.761+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f87c8059940 con 0x7f87d006c240 2024-03-08T08:29:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.976+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87d015b1c0 con 0x7f87d006c240 2024-03-08T08:29:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.976+0000 7f87ceffd640 1 -- 172.21.15.12:0/1005360488 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f87c8059240 con 0x7f87d006c240 2024-03-08T08:29:21.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:21.978 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 msgr2=0x7f87bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 0x7f87bc06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f87d01559e0 tx=0x7f87c4008040 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 msgr2=0x7f87d00cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f87c8007e40 tx=0x7f87c800e4b0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 shutdown_connections 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87d00b9420 0x7f87d00b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87d00b5550 0x7f87d01f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 --2- 172.21.15.12:0/1005360488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87d006c240 0x7f87d00cab10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 >> 172.21.15.12:0/1005360488 conn(0x7f87d0072900 msgr2=0x7f87d00ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 shutdown_connections 2024-03-08T08:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:21.978+0000 7f87d87b6640 1 -- 172.21.15.12:0/1005360488 wait complete. 2024-03-08T08:29:21.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:22.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.090+0000 7f0888888640 1 Processor -- start 2024-03-08T08:29:22.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.090+0000 7f0888888640 1 -- start start 2024-03-08T08:29:22.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0888888640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088406c240 0x7f088406c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0888888640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 0x7f088415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0888888640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088406cb60 con 0x7f08840b5550 2024-03-08T08:29:22.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0888888640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840b6b30 con 0x7f088406c240 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0888888640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088415ebc0 con 0x7f08840b9420 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088406c240 0x7f088406c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088406c240 0x7f088406c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38982/0 (socket says 172.21.15.12:38982) 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 -- 172.21.15.12:0/3532160437 learned_addr learned my addr 172.21.15.12:0/3532160437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0882d9d640 1 --1- 172.21.15.12:0/3532160437 >> v1:172.21.15.12:6789/0 conn(0x7f08840b5550 0x7f08840b59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33936/0 (socket says 172.21.15.12:33936) 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0881d9b640 1 --2- 172.21.15.12:0/3532160437 >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 0x7f088415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 -- 172.21.15.12:0/3532160437 >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 msgr2=0x7f088415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 --2- 172.21.15.12:0/3532160437 >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 0x7f088415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 -- 172.21.15.12:0/3532160437 >> v1:172.21.15.12:6789/0 conn(0x7f08840b5550 legacy=0x7f08840b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f088259c640 1 -- 172.21.15.12:0/3532160437 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088415eea0 con 0x7f088406c240 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.091+0000 7f0881d9b640 1 --2- 172.21.15.12:0/3532160437 >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 0x7f088415e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:22.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.092+0000 7f088259c640 1 --2- 172.21.15.12:0/3532160437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088406c240 0x7f088406c620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f088408cee0 tx=0x7f087c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c814cb907ce9e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.092+0000 7f088159a640 1 -- 172.21.15.12:0/3532160437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c02f850 con 0x7f088406c240 2024-03-08T08:29:22.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.092+0000 7f088159a640 1 -- 172.21.15.12:0/3532160437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c02f9f0 con 0x7f088406c240 2024-03-08T08:29:22.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.092+0000 7f088159a640 1 -- 172.21.15.12:0/3532160437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c02fdc0 con 0x7f088406c240 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 -- 172.21.15.12:0/3532160437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088406c240 msgr2=0x7f088406c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 --2- 172.21.15.12:0/3532160437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088406c240 0x7f088406c620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f088408cee0 tx=0x7f087c02ba30 comp rx=0 tx=0).stop 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 -- 172.21.15.12:0/3532160437 shutdown_connections 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 --2- 172.21.15.12:0/3532160437 >> v2:172.21.15.156:3300/0 conn(0x7f08840b9420 0x7f088415e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 --2- 172.21.15.12:0/3532160437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088406c240 0x7f088406c620 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 -- 172.21.15.12:0/3532160437 >> 172.21.15.12:0/3532160437 conn(0x7f0884072900 msgr2=0x7f08840d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 -- 172.21.15.12:0/3532160437 shutdown_connections 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.093+0000 7f0888888640 1 -- 172.21.15.12:0/3532160437 wait complete. 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.094+0000 7f0888888640 1 Processor -- start 2024-03-08T08:29:22.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.094+0000 7f0888888640 1 -- start start 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f088406c240 0x7f088415ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08840b5550 0x7f08841e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088415bc30 con 0x7f08840b9420 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840d1f90 con 0x7f08840b5550 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0888888640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088415f180 con 0x7f088406c240 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38992/0 (socket says 172.21.15.12:38992) 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 -- 172.21.15.12:0/3291499133 learned_addr learned my addr 172.21.15.12:0/3291499133 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f088259c640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f088406c240 0x7f088415ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0881d9b640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08840b5550 0x7f08841e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 -- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f088406c240 msgr2=0x7f088415ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f088406c240 0x7f088415ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 -- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08840b5550 msgr2=0x7f08841e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08840b5550 0x7f08841e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 -- 172.21.15.12:0/3291499133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08841e8e20 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.095+0000 7f0882d9d640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f087400c7b0 tx=0x7f087400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.096+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874017070 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.096+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087400a640 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.096+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08840cd7a0 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.096+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400a9f0 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.096+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08840cba50 con 0x7f08840b9420 2024-03-08T08:29:22.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.097+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08840749c0 con 0x7f08840b9420 2024-03-08T08:29:22.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.097+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f087401a070 con 0x7f08840b9420 2024-03-08T08:29:22.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.098+0000 7f087affd640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 0x7f086406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.098+0000 7f088259c640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 0x7f086406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.098+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0874089780 con 0x7f08840b9420 2024-03-08T08:29:22.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.098+0000 7f088259c640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 0x7f086406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f087c036000 tx=0x7f087c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.102+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0874057f10 con 0x7f08840b9420 2024-03-08T08:29:22.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.286+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f088415ddd0 con 0x7f08840b9420 2024-03-08T08:29:22.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.288+0000 7f087affd640 1 -- 172.21.15.12:0/3291499133 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f087405bfb0 con 0x7f08840b9420 2024-03-08T08:29:22.289 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:22.289 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 msgr2=0x7f086406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 0x7f086406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f087c036000 tx=0x7f087c008030 comp rx=0 tx=0).stop 2024-03-08T08:29:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 msgr2=0x7f08841e6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f087400c7b0 tx=0x7f087400cc70 comp rx=0 tx=0).stop 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 shutdown_connections 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086406b4d0 0x7f086406d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08840b9420 0x7f08841e6490 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08840b5550 0x7f08841e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 --2- 172.21.15.12:0/3291499133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f088406c240 0x7f088415ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 >> 172.21.15.12:0/3291499133 conn(0x7f0884072900 msgr2=0x7f08840b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 shutdown_connections 2024-03-08T08:29:22.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.290+0000 7f0888888640 1 -- 172.21.15.12:0/3291499133 wait complete. 2024-03-08T08:29:22.302 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:22.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.402+0000 7f729643b640 1 Processor -- start 2024-03-08T08:29:22.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.402+0000 7f729643b640 1 -- start start 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f729643b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72900b0be0 0x7f729015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f729643b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f729006abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f729643b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900cd3b0 con 0x7f729006b110 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f729643b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729006b560 con 0x7f72900b0be0 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f729643b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900cd690 con 0x7f72900b0fc0 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f728ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72900b0be0 0x7f729015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f728ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72900b0be0 0x7f729015c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39006/0 (socket says 172.21.15.12:39006) 2024-03-08T08:29:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.403+0000 7f728ffff640 1 -- 172.21.15.12:0/2894348059 learned_addr learned my addr 172.21.15.12:0/2894348059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:22.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728f7fe640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f729006abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728ffff640 1 -- 172.21.15.12:0/2894348059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 msgr2=0x7f729006abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:22.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728ffff640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f729006abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728ffff640 1 -- 172.21.15.12:0/2894348059 >> v1:172.21.15.156:6789/0 conn(0x7f729006b110 legacy=0x7f72900d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:22.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728ffff640 1 -- 172.21.15.12:0/2894348059 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900cd970 con 0x7f72900b0be0 2024-03-08T08:29:22.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.404+0000 7f728ffff640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f729015c9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7280009870 tx=0x7f728002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1365a796202488e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f728effd640 1 -- 172.21.15.12:0/2894348059 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280035020 con 0x7f72900b0be0 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f728effd640 1 -- 172.21.15.12:0/2894348059 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7280031640 con 0x7f72900b0be0 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f728effd640 1 -- 172.21.15.12:0/2894348059 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728002d040 con 0x7f72900b0be0 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 -- 172.21.15.12:0/2894348059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 msgr2=0x7f729015c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f729015c9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7280009870 tx=0x7f728002b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 -- 172.21.15.12:0/2894348059 shutdown_connections 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f729006abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 --2- 172.21.15.12:0/2894348059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f729015c9a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 -- 172.21.15.12:0/2894348059 >> 172.21.15.12:0/2894348059 conn(0x7f7290099a40 msgr2=0x7f7290099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 -- 172.21.15.12:0/2894348059 shutdown_connections 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.405+0000 7f729643b640 1 -- 172.21.15.12:0/2894348059 wait complete. 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.406+0000 7f729643b640 1 Processor -- start 2024-03-08T08:29:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.406+0000 7f729643b640 1 -- start start 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f729006b110 0x7f72900ad710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f72901f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b18c0 con 0x7f72900b0be0 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900cdf30 con 0x7f729006b110 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f729643b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729015cee0 con 0x7f72900b0fc0 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39016/0 (socket says 172.21.15.12:39016) 2024-03-08T08:29:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.407+0000 7f728f7fe640 1 -- 172.21.15.12:0/855660737 learned_addr learned my addr 172.21.15.12:0/855660737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728ffff640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f729006b110 0x7f72900ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f72949d8640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f72901f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 -- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 msgr2=0x7f72901f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f72901f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 -- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f729006b110 msgr2=0x7f72900ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f729006b110 0x7f72900ad710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 -- 172.21.15.12:0/855660737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290143d30 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728f7fe640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f728400c820 tx=0x7f728400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284015070 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7284004020 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72840043f0 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900add20 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.408+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900b0170 con 0x7f72900b0be0 2024-03-08T08:29:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.409+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729006b6e0 con 0x7f72900b0be0 2024-03-08T08:29:22.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.409+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7284007830 con 0x7f72900b0be0 2024-03-08T08:29:22.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.410+0000 7f728cff9640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:22.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.410+0000 7f728ffff640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:22.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.410+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7284089f30 con 0x7f72900b0be0 2024-03-08T08:29:22.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.410+0000 7f728ffff640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 0x7f727c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7280036000 tx=0x7f7280008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.414+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f729006b6e0 con 0x7f72900b0be0 2024-03-08T08:29:22.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.633+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72900749c0 con 0x7f72900b0be0 2024-03-08T08:29:22.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.633+0000 7f728cff9640 1 -- 172.21.15.12:0/855660737 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f7284058e40 con 0x7f72900b0be0 2024-03-08T08:29:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:22.634 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:22.635 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 msgr2=0x7f727c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 0x7f727c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7280036000 tx=0x7f7280008030 comp rx=0 tx=0).stop 2024-03-08T08:29:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 msgr2=0x7f72901437f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f728400c820 tx=0x7f728400cce0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 shutdown_connections 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72900b0fc0 0x7f72901f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72900b0be0 0x7f72901437f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 --2- 172.21.15.12:0/855660737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f729006b110 0x7f72900ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 >> 172.21.15.12:0/855660737 conn(0x7f7290099a40 msgr2=0x7f72900cbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 shutdown_connections 2024-03-08T08:29:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:22.635+0000 7f729643b640 1 -- 172.21.15.12:0/855660737 wait complete. 2024-03-08T08:29:23.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-08T08:29:05.596568+0000', 'modified': '2024-03-08T08:29:07.598513+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6837/3547610523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6835', 'nonce': 3547610523}, {'type': 'v1', 'addr': '172.21.15.12:6837', 'nonce': 3547610523}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-08T08:29:23.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-08T08:29:23.049 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-08T08:29:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.153+0000 7f9f2600e640 1 Processor -- start 2024-03-08T08:29:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.153+0000 7f9f2600e640 1 -- start start 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f2600e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f2015ddc0 0x7f9f200d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f2600e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2015e1a0 con 0x7f9f200d0f40 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f2600e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200b8790 con 0x7f9f200b7bf0 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f2600e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f200b9030 con 0x7f9f2015ddc0 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f2015ddc0 0x7f9f200d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9f200d0f40 0x7f9f200d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33946/0 (socket says 172.21.15.12:33946) 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f2015ddc0 0x7f9f200d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39024/0 (socket says 172.21.15.12:39024) 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2052072603 learned_addr learned my addr 172.21.15.12:0/2052072603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2052072603 >> v1:172.21.15.156:6789/0 conn(0x7f9f200b7bf0 legacy=0x7f9f200d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2052072603 >> v1:172.21.15.12:6789/0 conn(0x7f9f200d0f40 legacy=0x7f9f200d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2052072603 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200b8a70 con 0x7f9f2015ddc0 2024-03-08T08:29:23.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.154+0000 7f9f1e7fc640 1 -- 172.21.15.12:0/2052072603 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011040277 0 0) 0x7f9f2015e1a0 con 0x7f9f200d0f40 2024-03-08T08:29:23.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.155+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2052072603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200d0a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009870 tx=0x7f9f1002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9ab75c4e90cc17 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.155+0000 7f9f1e7fc640 1 -- 172.21.15.12:0/2052072603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1002f840 con 0x7f9f2015ddc0 2024-03-08T08:29:23.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.155+0000 7f9f1e7fc640 1 -- 172.21.15.12:0/2052072603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1002f9e0 con 0x7f9f2015ddc0 2024-03-08T08:29:23.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.155+0000 7f9f1e7fc640 1 -- 172.21.15.12:0/2052072603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1002fdb0 con 0x7f9f2015ddc0 2024-03-08T08:29:23.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 -- 172.21.15.12:0/2052072603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 msgr2=0x7f9f200d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 --2- 172.21.15.12:0/2052072603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200d0a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009870 tx=0x7f9f1002ba20 comp rx=0 tx=0).stop 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 -- 172.21.15.12:0/2052072603 shutdown_connections 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 --2- 172.21.15.12:0/2052072603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200d0a00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 -- 172.21.15.12:0/2052072603 >> 172.21.15.12:0/2052072603 conn(0x7f9f20072720 msgr2=0x7f9f20072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 -- 172.21.15.12:0/2052072603 shutdown_connections 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.156+0000 7f9f2600e640 1 -- 172.21.15.12:0/2052072603 wait complete. 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.157+0000 7f9f2600e640 1 Processor -- start 2024-03-08T08:29:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.157+0000 7f9f2600e640 1 -- start start 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f200b7bf0 0x7f9f20157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 0x7f9f2015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200b9980 con 0x7f9f2015ddc0 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200d1d20 con 0x7f9f200b7bf0 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f2600e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200d2fd0 con 0x7f9f200d0f40 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39034/0 (socket says 172.21.15.12:39034) 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 -- 172.21.15.12:0/3964694769 learned_addr learned my addr 172.21.15.12:0/3964694769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1effd640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 0x7f9f2015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f200b7bf0 0x7f9f20157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 -- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 msgr2=0x7f9f2015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 0x7f9f2015e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 -- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f200b7bf0 msgr2=0x7f9f20157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f200b7bf0 0x7f9f20157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 -- 172.21.15.12:0/3964694769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2015ece0 con 0x7f9f2015ddc0 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1effd640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 0x7f9f2015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c7a0 tx=0x7f9f1800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f07fff640 1 -- 172.21.15.12:0/3964694769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18003b70 con 0x7f9f2015ddc0 2024-03-08T08:29:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f07fff640 1 -- 172.21.15.12:0/3964694769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18003d30 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.158+0000 7f9f07fff640 1 -- 172.21.15.12:0/3964694769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1800aa90 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.159+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2015efc0 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.159+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2015f7c0 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.160+0000 7f9f07fff640 1 -- 172.21.15.12:0/3964694769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f1800ac30 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.160+0000 7f9f07fff640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.160+0000 7f9f07fff640 1 -- 172.21.15.12:0/3964694769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f1808a120 con 0x7f9f2015ddc0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.160+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.161+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f10036000 tx=0x7f9f10008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.161+0000 7f9f28bef740 1 Processor -- start 2024-03-08T08:29:23.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.161+0000 7f9f28bef740 1 -- start start 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e1006b8c0 0x561e1006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10097600 0x561e10097a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e0ffd16b0 con 0x561e10067870 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e0ff7aa60 con 0x561e1006b8c0 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e0ffcd750 con 0x561e10097600 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39044/0 (socket says 172.21.15.12:39044) 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.162+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/4165546046 learned_addr learned my addr 172.21.15.12:0/4165546046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10097600 msgr2=0x561e10097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10097600 0x561e10097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1effd640 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e1006b8c0 0x561e1006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e1006b8c0 msgr2=0x561e1006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e1006b8c0 0x561e1006d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/4165546046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e0ff98f30 con 0x561e10067870 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009960 tx=0x7f9f10030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98471cb6684217ff server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f057fa640 1 -- 172.21.15.12:0/4165546046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10042b70 con 0x561e10067870 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f057fa640 1 -- 172.21.15.12:0/4165546046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10051070 con 0x561e10067870 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f057fa640 1 -- 172.21.15.12:0/4165546046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10037400 con 0x561e10067870 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.163+0000 7f9f28bef740 1 -- 172.21.15.12:0/4165546046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 msgr2=0x561e10067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009960 tx=0x7f9f10030000 comp rx=0 tx=0).stop 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 -- 172.21.15.12:0/4165546046 shutdown_connections 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10097600 0x561e10097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e1006b8c0 0x561e1006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 --2- 172.21.15.12:0/4165546046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10067870 0x561e10067c50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 -- 172.21.15.12:0/4165546046 >> 172.21.15.12:0/4165546046 conn(0x561e100698f0 msgr2=0x561e1006b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 -- 172.21.15.12:0/4165546046 shutdown_connections 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 -- 172.21.15.12:0/4165546046 wait complete. 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 -- start start 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.164+0000 7f9f28bef740 1 Processor -- start 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e100928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e10098db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1006bd50 con 0x561e10097600 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e0ff8dea0 con 0x561e10098930 2024-03-08T08:29:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e1006c030 con 0x561e1009afe0 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f28bef740 10 client.0 _ll_register_callbacks cb 0x561e0ff8d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39048/0 (socket says 172.21.15.12:39048) 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2017047069 learned_addr learned my addr 172.21.15.12:0/2017047069 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e100928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 msgr2=0x561e100928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e100928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 msgr2=0x561e10098db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e10098db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e10098db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.165+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2017047069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e0ff8e270 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f10031010 tx=0x7f9f10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb88b3012948935 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10003810 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f100039b0 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f10051070 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f10003bb0 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 10 client.4692 fetch_fsmap learned FSMap version 6 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561e0fce62d0 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10051070 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9f1005f430 con 0x561e10097600 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 10 client.4692 fetch_fsmap finished waiting for FSMap version 6 2024-03-08T08:29:23.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 10 client.4692 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 20 client.4692 populate_metadata read hostname 'smithi012' 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10099be0 0x561e100c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 4 client.4692 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 --> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e0ffdebf0 con 0x561e10099be0 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.166+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10099be0 0x561e100c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9efe7fc640 10 client.4692 ms_handle_connect on v2:172.21.15.12:6835/3547610523 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10099be0 0x561e100c28f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5057c1aaa17993cb server_cookie=a7df3d3642dd0cfe in_seq=0 out_seq=0 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2017047069 <== mds.0 v2:172.21.15.12:6835/3547610523 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561e0ffdebf0 con 0x561e10099be0 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9efe7fc640 10 client.4692 handle_command_reply: tid=0 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 client.4692 shutdown 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 msgr2=0x561e100b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f10031010 tx=0x7f9f10004730 comp rx=0 tx=0).stop 2024-03-08T08:29:23.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 shutdown_connections 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e10098db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10099be0 0x561e100c28f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e100928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2017047069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e100b02b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 >> 172.21.15.12:0/2017047069 conn(0x561e10067870 msgr2=0x561e100955c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9efe7fc640 0 client.4692 ms_handle_reset on v2:172.21.15.12:6835/3547610523 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 shutdown_connections 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.167+0000 7f9f28bef740 1 -- 172.21.15.12:0/2017047069 wait complete. 2024-03-08T08:29:23.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.168+0000 7f9f28bef740 20 client.4692 trim_cache size 0 max 16384 2024-03-08T08:29:23.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.183+0000 7f9f28bef740 1 Processor -- start 2024-03-08T08:29:23.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.183+0000 7f9f28bef740 1 -- start start 2024-03-08T08:29:23.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 0x561e10092680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10098930 0x561e100af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201fef40 con 0x561e1009afe0 2024-03-08T08:29:23.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201f77d0 con 0x561e10097600 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20160a70 con 0x561e10098930 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39058/0 (socket says 172.21.15.12:39058) 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 -- 172.21.15.12:0/402602625 learned_addr learned my addr 172.21.15.12:0/402602625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 -- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10098930 msgr2=0x561e100af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 0x561e10092680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10098930 0x561e100af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 -- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 msgr2=0x561e10092680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 0x561e10092680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1ffff640 1 -- 172.21.15.12:0/402602625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e0fdb06b0 con 0x561e1009afe0 2024-03-08T08:29:23.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.184+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 0x561e10092680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9f18060a70 tx=0x7f9f1801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e02f05d8ddaaa25 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f04ff9640 1 -- 172.21.15.12:0/402602625 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18058f90 con 0x561e1009afe0 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f04ff9640 1 -- 172.21.15.12:0/402602625 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18058890 con 0x561e1009afe0 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f04ff9640 1 -- 172.21.15.12:0/402602625 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1805c9e0 con 0x561e1009afe0 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 -- 172.21.15.12:0/402602625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 msgr2=0x561e100af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9f18060a70 tx=0x7f9f1801b000 comp rx=0 tx=0).stop 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 -- 172.21.15.12:0/402602625 shutdown_connections 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e1009afe0 0x561e100af820 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e10098930 0x561e100af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 --2- 172.21.15.12:0/402602625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10097600 0x561e10092680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 -- 172.21.15.12:0/402602625 >> 172.21.15.12:0/402602625 conn(0x561e10067870 msgr2=0x561e1006b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 -- 172.21.15.12:0/402602625 shutdown_connections 2024-03-08T08:29:23.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.185+0000 7f9f28bef740 1 -- 172.21.15.12:0/402602625 wait complete. 2024-03-08T08:29:23.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.186+0000 7f9f28bef740 1 -- start start 2024-03-08T08:29:23.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.186+0000 7f9f28bef740 1 Processor -- start 2024-03-08T08:29:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e1006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e1006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e100920c0 con 0x561e10097600 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1009b7c0 con 0x561e10098930 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e100c6480 con 0x561e1009afe0 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f28bef740 10 client.0 _ll_register_callbacks cb 0x561e100680e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39070/0 (socket says 172.21.15.12:39070) 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2942403607 learned_addr learned my addr 172.21.15.12:0/2942403607 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e1006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e1006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 msgr2=0x561e1006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e1006c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 msgr2=0x561e1006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e1006bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 -- 172.21.15.12:0/2942403607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e1006cde0 con 0x561e10097600 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1ffff640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e1006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9f1f7fe640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006b00 tx=0x7f9f10009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7889935d3e66998 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9efe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.187+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10070060 con 0x561e10097600 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f10004350 con 0x561e10097600 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f18058890 con 0x561e10097600 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f10042840 con 0x561e10097600 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 10 client.4698 fetch_fsmap learned FSMap version 6 2024-03-08T08:29:23.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561e100c44e0 con 0x561e10097600 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18058890 con 0x561e10097600 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9f10003610 con 0x561e10097600 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 10 client.4698 fetch_fsmap finished waiting for FSMap version 6 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 10 client.4698 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 20 client.4698 populate_metadata read hostname 'smithi012' 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10096e20 0x561e100b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 4 client.4698 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 --> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561e0ffd8f40 con 0x561e10096e20 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10096e20 0x561e100b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 10 client.4698 ms_handle_connect on v2:172.21.15.12:6835/3547610523 2024-03-08T08:29:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f1effd640 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10096e20 0x561e100b1bc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9b49dde7b023fad server_cookie=cac2ff0464dfa06 in_seq=0 out_seq=0 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 1 -- 172.21.15.12:0/2942403607 <== mds.0 v2:172.21.15.12:6835/3547610523 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x561e0ffd8f40 con 0x561e10096e20 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9efe7fc640 10 client.4698 handle_command_reply: tid=0 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.188+0000 7f9f28bef740 1 client.4698 shutdown 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 msgr2=0x561e10094430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006b00 tx=0x7f9f10009240 comp rx=0 tx=0).stop 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 shutdown_connections 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:6835/3547610523,v1:172.21.15.12:6837/3547610523] conn(0x561e10096e20 0x561e100b1bc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561e1009afe0 0x561e1006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e10098930 0x561e1006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9efe7fc640 0 client.4698 ms_handle_reset on v2:172.21.15.12:6835/3547610523 2024-03-08T08:29:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 --2- 172.21.15.12:0/2942403607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e10097600 0x561e10094430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 >> 172.21.15.12:0/2942403607 conn(0x561e10067870 msgr2=0x561e100925c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 shutdown_connections 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 1 -- 172.21.15.12:0/2942403607 wait complete. 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f28bef740 20 client.4698 trim_cache size 0 max 16384 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c", 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4254, 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-08T08:29:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 16.957917531, 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 6, 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 21, 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 21, 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 23.913028848 2024-03-08T08:29:23.200 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 msgr2=0x7f9f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f10036000 tx=0x7f9f10008030 comp rx=0 tx=0).stop 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 msgr2=0x7f9f200ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c7a0 tx=0x7f9f1800cc60 comp rx=0 tx=0).stop 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 shutdown_connections 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f2015ddc0 0x7f9f200ccb90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f200d0f40 0x7f9f2015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 --2- 172.21.15.12:0/3964694769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f200b7bf0 0x7f9f20157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 >> 172.21.15.12:0/3964694769 conn(0x7f9f20072720 msgr2=0x7f9f200ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 shutdown_connections 2024-03-08T08:29:23.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.189+0000 7f9f2600e640 1 -- 172.21.15.12:0/3964694769 wait complete. 2024-03-08T08:29:23.203 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-08T08:29:23.226 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-08T08:29:23.226 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-08T08:29:23.226 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-08T08:29:23.226 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T08:29:23.226 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:23.226 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-08T08:29:23.226 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T08:29:23.226 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:23.226 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-08T08:29:23.227 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T08:29:23.227 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:23.227 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-08T08:29:23.227 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T08:29:23.227 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:23.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.227 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-08T08:29:23.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-08T08:29:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-03-08T08:29:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:29:23.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.312 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-08T08:29:23.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.330 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-03-08T08:29:23.403 INFO:teuthology.orchestra.run.smithi156.stderr:Cannot find device "ceph-brx" 2024-03-08T08:29:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:29:23.405 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-08T08:29:23.405 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:29:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.506+0000 7fdb114bb640 1 Processor -- start 2024-03-08T08:29:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.506+0000 7fdb114bb640 1 -- start start 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb114bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb0c0b3910 0x7fdb0c070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb114bb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c0af330 con 0x7fdb0c0aef50 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb114bb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c15e270 con 0x7fdb0c06e690 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb114bb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c070680 con 0x7fdb0c0b3910 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0a7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb0c0b3910 0x7fdb0c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdb0c0aef50 0x7fdb0c15ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33962/0 (socket says 172.21.15.12:33962) 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0a7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb0c0b3910 0x7fdb0c070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39080/0 (socket says 172.21.15.12:39080) 2024-03-08T08:29:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0affd640 1 -- 172.21.15.12:0/4021826874 learned_addr learned my addr 172.21.15.12:0/4021826874 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0a7fc640 1 -- 172.21.15.12:0/4021826874 >> v1:172.21.15.156:6789/0 conn(0x7fdb0c06e690 legacy=0x7fdb0c06abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0a7fc640 1 -- 172.21.15.12:0/4021826874 >> v1:172.21.15.12:6789/0 conn(0x7fdb0c0aef50 legacy=0x7fdb0c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb0a7fc640 1 -- 172.21.15.12:0/4021826874 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0b3d90 con 0x7fdb0c0b3910 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.507+0000 7fdb09ffb640 1 -- 172.21.15.12:0/4021826874 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677660913 0 0) 0x7fdafc001110 con 0x7fdb0c0aef50 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb0a7fc640 1 --2- 172.21.15.12:0/4021826874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c070140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb000097a0 tx=0x7fdb0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a0e6cd249df0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb09ffb640 1 -- 172.21.15.12:0/4021826874 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb00034070 con 0x7fdb0c0b3910 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb09ffb640 1 -- 172.21.15.12:0/4021826874 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb00030610 con 0x7fdb0c0b3910 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb09ffb640 1 -- 172.21.15.12:0/4021826874 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000309e0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 -- 172.21.15.12:0/4021826874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 msgr2=0x7fdb0c070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 --2- 172.21.15.12:0/4021826874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c070140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb000097a0 tx=0x7fdb0002b960 comp rx=0 tx=0).stop 2024-03-08T08:29:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 -- 172.21.15.12:0/4021826874 shutdown_connections 2024-03-08T08:29:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 --2- 172.21.15.12:0/4021826874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c070140 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 -- 172.21.15.12:0/4021826874 >> 172.21.15.12:0/4021826874 conn(0x7fdb0c0b5ca0 msgr2=0x7fdb0c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 -- 172.21.15.12:0/4021826874 shutdown_connections 2024-03-08T08:29:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.508+0000 7fdb114bb640 1 -- 172.21.15.12:0/4021826874 wait complete. 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 Processor -- start 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 -- start start 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0c06e690 0x7fdb0c1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdb0c0aef50 0x7fdb0c1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c15e4f0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0b58f0 con 0x7fdb0c06e690 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.510+0000 7fdb114bb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c15f120 con 0x7fdb0c0aef50 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39092/0 (socket says 172.21.15.12:39092) 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 -- 172.21.15.12:0/1304238703 learned_addr learned my addr 172.21.15.12:0/1304238703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0affd640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0c06e690 0x7fdb0c1f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0a7fc640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdb0c0aef50 0x7fdb0c1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 -- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdb0c0aef50 msgr2=0x7fdb0c1f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdb0c0aef50 0x7fdb0c1f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 -- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0c06e690 msgr2=0x7fdb0c1f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0c06e690 0x7fdb0c1f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 -- 172.21.15.12:0/1304238703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c1f46a0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb0b7fe640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c7a0 tx=0x7fdb0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04003b70 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb04003d30 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c0cc7c0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400aa90 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.511+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c0b2db0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.512+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c0749c0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.513+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb0400ac30 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.513+0000 7fdaf37fe640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 0x7fdaf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.513+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb04089f70 con 0x7fdb0c0b3910 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.513+0000 7fdb0affd640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 0x7fdaf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.513+0000 7fdb0affd640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 0x7fdaf806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000f60 tx=0x7fdafc007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.517+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdb0c1f46a0 con 0x7fdb0c0b3910 2024-03-08T08:29:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.741+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdb0c0b2880 con 0x7fdb0c0b3910 2024-03-08T08:29:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.741+0000 7fdaf37fe640 1 -- 172.21.15.12:0/1304238703 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdb0c15e270 con 0x7fdb0c0b3910 2024-03-08T08:29:23.743 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 msgr2=0x7fdaf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 0x7fdaf806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000f60 tx=0x7fdafc007b50 comp rx=0 tx=0).stop 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 msgr2=0x7fdb0c15d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c7a0 tx=0x7fdb0400cc60 comp rx=0 tx=0).stop 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 shutdown_connections 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdaf806b4d0 0x7fdaf806d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb0c0b3910 0x7fdb0c15d080 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdb0c0aef50 0x7fdb0c1f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 --2- 172.21.15.12:0/1304238703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0c06e690 0x7fdb0c1f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.743+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 >> 172.21.15.12:0/1304238703 conn(0x7fdb0c0b5ca0 msgr2=0x7fdb0c15b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.744+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 shutdown_connections 2024-03-08T08:29:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:23.744+0000 7fdb114bb640 1 -- 172.21.15.12:0/1304238703 wait complete. 2024-03-08T08:29:23.755 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:29:23.755 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:29:23.755 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-08T08:29:23.755 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-08T08:29:23.756 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:23.756 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:29:23.756 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:29:23.756 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:29:23.756 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-08T08:29:23.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.756 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-08T08:29:23.770 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:23.770 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:23.770 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:23.771 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-08T08:29:23.772 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-08T08:29:23.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-08T08:29:23.773 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:23.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.930 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:29:23.956 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-08T08:29:23.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:23.958 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:29:24.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-08T08:29:24.039 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:24.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.293 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-08T08:29:24.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.308 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-08T08:29:24.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.363 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:24.363 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:24.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:24.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-08T08:29:24.363 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:24.445 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-08T08:29:24.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-08T08:29:24.446 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:24.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.592 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:24.592 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:24.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-08T08:29:24.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-03-08T08:29:24.592 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:24.634 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-08T08:29:24.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:24.634 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:24.687 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-08T08:29:24.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:24.687 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:24.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:29:24.809 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:29:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:29:24.866 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:29:24.867 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:29:24.867 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:24.867 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:24.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:24.867 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:29:24.919 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-08T08:29:24.919 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-08T08:29:24.961 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:29:24.987 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:29:25.043 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:29:25.044 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:25.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.045 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:29:25.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-03-08T08:29:25.047+0000 7f923075a480 -1 init, newargv = 0x55f4e2d62500 newargc=15 2024-03-08T08:29:25.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[46103]: starting ceph client 2024-03-08T08:29:25.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[46103]: starting fuse 2024-03-08T08:29:25.063 INFO:teuthology.orchestra.run.smithi012.stdout:41 2024-03-08T08:29:25.063 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-08T08:29:25.063 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:25.064 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-03-08T08:29:25.065 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-08T08:29:25.184 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.46103.asok 2024-03-08T08:29:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:29:25.195 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46103.asok 2024-03-08T08:29:25.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46103.asok status 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "46103", 2024-03-08T08:29:25.334 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4713, 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4713 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3142253620 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:29:25.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3142253620 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.4713 192.168.0.1:0/3142253620", 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/3142253620", 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 6, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 21, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-03-08T08:29:25.336 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:29:25.343 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:29:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.443+0000 7f94d446b640 1 Processor -- start 2024-03-08T08:29:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.443+0000 7f94d446b640 1 -- start start 2024-03-08T08:29:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d446b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d446b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 0x7f94cc0cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d446b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc0b4420 con 0x7f94cc143ec0 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d446b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc06f070 con 0x7f94cc06eb30 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d446b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc15e510 con 0x7f94cc143ae0 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d2207640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d2207640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:54466/0 (socket says 172.21.15.12:54466) 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d2207640 1 -- 172.21.15.12:0/2712697669 learned_addr learned my addr 172.21.15.12:0/2712697669 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.444+0000 7f94d1a06640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 0x7f94cc0cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d2207640 1 -- 172.21.15.12:0/2712697669 >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 msgr2=0x7f94cc0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d2207640 1 --2- 172.21.15.12:0/2712697669 >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 0x7f94cc0cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d2207640 1 -- 172.21.15.12:0/2712697669 >> v1:172.21.15.156:6789/0 conn(0x7f94cc143ec0 legacy=0x7f94cc0714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d2207640 1 -- 172.21.15.12:0/2712697669 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc15e7f0 con 0x7f94cc143ae0 2024-03-08T08:29:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d1a06640 1 --2- 172.21.15.12:0/2712697669 >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 0x7f94cc0cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.445+0000 7f94d2207640 1 --2- 172.21.15.12:0/2712697669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94c800bb10 tx=0x7f94c802e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e544236023d786e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d1205640 1 -- 172.21.15.12:0/2712697669 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c803b070 con 0x7f94cc143ae0 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d1205640 1 -- 172.21.15.12:0/2712697669 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c8036440 con 0x7f94cc143ae0 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d1205640 1 -- 172.21.15.12:0/2712697669 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8036810 con 0x7f94cc143ae0 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 -- 172.21.15.12:0/2712697669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 msgr2=0x7f94cc06e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 --2- 172.21.15.12:0/2712697669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94c800bb10 tx=0x7f94c802e150 comp rx=0 tx=0).stop 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 -- 172.21.15.12:0/2712697669 shutdown_connections 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 --2- 172.21.15.12:0/2712697669 >> v2:172.21.15.156:3300/0 conn(0x7f94cc06eb30 0x7f94cc0cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 --2- 172.21.15.12:0/2712697669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc06e5f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 -- 172.21.15.12:0/2712697669 >> 172.21.15.12:0/2712697669 conn(0x7f94cc06abd0 msgr2=0x7f94cc06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:25.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 -- 172.21.15.12:0/2712697669 shutdown_connections 2024-03-08T08:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.446+0000 7f94d446b640 1 -- 172.21.15.12:0/2712697669 wait complete. 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.447+0000 7f94d446b640 1 Processor -- start 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.447+0000 7f94d446b640 1 -- start start 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc143ec0 0x7f94cc13b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc15f2d0 con 0x7f94cc06eb30 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc15edb0 con 0x7f94cc143ec0 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d446b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc144560 con 0x7f94cc143ae0 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39100/0 (socket says 172.21.15.12:39100) 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 -- 172.21.15.12:0/2389667070 learned_addr learned my addr 172.21.15.12:0/2389667070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d1a06640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2a08640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc143ec0 0x7f94cc13b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 -- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 msgr2=0x7f94cc159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 -- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc143ec0 msgr2=0x7f94cc13b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc143ec0 0x7f94cc13b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.448+0000 7f94d2207640 1 -- 172.21.15.12:0/2389667070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc13c380 con 0x7f94cc06eb30 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94d1a06640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94d2207640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f94c802e190 tx=0x7f94c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c803b030 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c803d050 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc13c660 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8034870 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.449+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc13cbe0 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.450+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc13ce70 con 0x7f94cc06eb30 2024-03-08T08:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.450+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94c8048060 con 0x7f94cc06eb30 2024-03-08T08:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.451+0000 7f94c2ffd640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.451+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94c80aba70 con 0x7f94cc06eb30 2024-03-08T08:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.451+0000 7f94d1a06640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.451+0000 7f94d1a06640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 0x7f94b006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94cc13f7c0 tx=0x7f94bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.455+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f94c807a330 con 0x7f94cc06eb30 2024-03-08T08:29:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.673+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f94cc0749c0 con 0x7f94cc06eb30 2024-03-08T08:29:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.673+0000 7f94c2ffd640 1 -- 172.21.15.12:0/2389667070 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f94c807e3d0 con 0x7f94cc06eb30 2024-03-08T08:29:25.674 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 msgr2=0x7f94b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 0x7f94b006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94cc13f7c0 tx=0x7f94bc008040 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 msgr2=0x7f94cc13a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f94c802e190 tx=0x7f94c80046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 shutdown_connections 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc143ec0 0x7f94cc13b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94cc143ae0 0x7f94cc159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 --2- 172.21.15.12:0/2389667070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94cc06eb30 0x7f94cc13a3b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 >> 172.21.15.12:0/2389667070 conn(0x7f94cc06abd0 msgr2=0x7f94cc13e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 shutdown_connections 2024-03-08T08:29:25.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:25.675+0000 7f94d446b640 1 -- 172.21.15.12:0/2389667070 wait complete. 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:29:25.687 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-08T08:29:25.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.688 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42607sec preferred_lft 42607sec 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 76:bf:65:0f:db:fc brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::a4b2:1eff:fe40:f0ec/64 scope link 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 76:bf:65:0f:db:fc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::74bf:65ff:fe0f:dbfc/64 scope link 2024-03-08T08:29:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:25.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.705 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:29:25.766 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-08T08:29:25.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.768 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-08T08:29:25.824 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-08T08:29:25.824 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:29:25.824 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-08T08:29:25.824 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-08T08:29:25.825 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:29:25.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-08T08:29:25.825 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:25.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.919 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-08T08:29:25.933 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-08T08:29:25.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.934 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-08T08:29:25.988 INFO:teuthology.orchestra.run.smithi012.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-08T08:29:25.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:25.988 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:25.988 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:25.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:29:25.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-08T08:29:25.988 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:26.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether de:5c:1b:31:6b:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-08T08:29:26.135 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-08T08:29:26.136 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.136 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::dc5c:1bff:fe31:6be0/64 scope link 2024-03-08T08:29:26.136 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether de:5c:1b:31:6b:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.203 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::dc5c:1bff:fe31:6be0/64 scope link 2024-03-08T08:29:26.204 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:26.204 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-08T08:29:26.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-08T08:29:26.205 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:26.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.329 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:29:26.329 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:29:26.330 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.1 up 2024-03-08T08:29:26.330 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.1 master ceph-brx 2024-03-08T08:29:26.330 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:29:26.370 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-08T08:29:26.370 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:26.371 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:26.425 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-08T08:29:26.426 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:26.426 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:26.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:29:26.546 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:29:26.602 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:26.603 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:29:26.604 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:29:26.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.605 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:29:26.656 INFO:teuthology.orchestra.run.smithi012.stdout:41 2024-03-08T08:29:26.656 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-08T08:29:26.656 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-08T08:29:26.698 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:29:26.724 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:29:26.780 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:26.781 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:29:26.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.783 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:29:26.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:2024-03-08T08:29:26.781+0000 7f6eab44c480 -1 init, newargv = 0x56361fbb24f0 newargc=15 2024-03-08T08:29:26.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[46598]: starting ceph client 2024-03-08T08:29:26.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[46598]: starting fuse 2024-03-08T08:29:26.797 INFO:teuthology.orchestra.run.smithi012.stdout:41 2024-03-08T08:29:26.797 INFO:teuthology.orchestra.run.smithi012.stdout:44 2024-03-08T08:29:26.798 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-08T08:29:26.798 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.2")) 2024-03-08T08:29:26.799 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-08T08:29:26.918 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.2.46598.asok 2024-03-08T08:29:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:29:26.928 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.46598.asok 2024-03-08T08:29:26.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:26.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.46598.asok status 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "2", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "46598", 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:27.066 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4725, 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4725 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1004922560 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:29:27.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1004922560 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.4725 192.168.0.2:0/1004922560", 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.2:0/1004922560", 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 6, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 21, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-03-08T08:29:27.068 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:29:27.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:29:27.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.174+0000 7f606b381640 1 Processor -- start 2024-03-08T08:29:27.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.174+0000 7f606b381640 1 -- start start 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606b381640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f606406c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606b381640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f60640b9420 0x7f606415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606b381640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606406cb60 con 0x7f60640b5550 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606b381640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60640b6b30 con 0x7f60640b9420 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606b381640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606415ebc0 con 0x7f606406c240 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606991e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f60640b5550 0x7f60640b5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33970/0 (socket says 172.21.15.12:33970) 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606991e640 1 -- 172.21.15.12:0/1317565487 learned_addr learned my addr 172.21.15.12:0/1317565487 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f606406c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f606891c640 1 --2- 172.21.15.12:0/1317565487 >> v2:172.21.15.156:3300/0 conn(0x7f60640b9420 0x7f606415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157910523 0 0) 0x7f606406cb60 con 0x7f60640b5550 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c003520 con 0x7f60640b5550 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.175+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981313659 0 0) 0x7f604c003520 con 0x7f60640b5550 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 msgr2=0x7f606406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f606406c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 >> v2:172.21.15.156:3300/0 conn(0x7f60640b9420 msgr2=0x7f606415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 --2- 172.21.15.12:0/1317565487 >> v2:172.21.15.156:3300/0 conn(0x7f60640b9420 0x7f606415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606415eea0 con 0x7f60640b5550 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f6054001670 con 0x7f60640b5550 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f606406c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 >> v1:172.21.15.12:6789/0 conn(0x7f60640b5550 legacy=0x7f60640b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c0039b0 0x7f604c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c004800 0x7f604c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f604c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606415ebc0 con 0x7f604c0039b0 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c00b9c0 con 0x7f604c004800 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c00bb40 con 0x7f606406c240 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c0039b0 0x7f604c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606891c640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c004800 0x7f604c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606991e640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f604c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 -- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 msgr2=0x7f604c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f606406c240 0x7f604c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 -- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c004800 msgr2=0x7f604c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c004800 0x7f604c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.176+0000 7f606911d640 1 -- 172.21.15.12:0/1317565487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c003520 con 0x7f604c0039b0 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606911d640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c0039b0 0x7f604c003dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f605c00e350 tx=0x7f604c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1771f5609781a918 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c039070 con 0x7f604c0039b0 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c02cd60 con 0x7f604c0039b0 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f6063fff640 1 -- 172.21.15.12:0/1317565487 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c009de0 con 0x7f604c0039b0 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606b381640 1 -- 172.21.15.12:0/1317565487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c0039b0 msgr2=0x7f604c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606b381640 1 --2- 172.21.15.12:0/1317565487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c0039b0 0x7f604c003dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f605c00e350 tx=0x7f604c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606991e640 1 -- 172.21.15.12:0/1317565487 reap_dead start 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606b381640 1 -- 172.21.15.12:0/1317565487 shutdown_connections 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606b381640 1 -- 172.21.15.12:0/1317565487 >> 172.21.15.12:0/1317565487 conn(0x7f6064072900 msgr2=0x7f60640d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.177+0000 7f606b381640 1 -- 172.21.15.12:0/1317565487 shutdown_connections 2024-03-08T08:29:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.178+0000 7f606b381640 1 -- 172.21.15.12:0/1317565487 wait complete. 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 Processor -- start 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 -- start start 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 0x7f60640b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 0x7f60640caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606415bc30 con 0x7f60641512d0 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60640d1f90 con 0x7f60640b7b10 2024-03-08T08:29:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606b381640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606415f180 con 0x7f60640cab10 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.179+0000 7f606911d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39108/0 (socket says 172.21.15.12:39108) 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 -- 172.21.15.12:0/1587455042 learned_addr learned my addr 172.21.15.12:0/1587455042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606891c640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 0x7f60640b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606991e640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 0x7f60640caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 -- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 msgr2=0x7f60640caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 0x7f60640caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 -- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 msgr2=0x7f60640b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 0x7f60640b7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 -- 172.21.15.12:0/1587455042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640cdb20 con 0x7f60641512d0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606991e640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 0x7f60640caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606891c640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 0x7f60640b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606911d640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f605c00e470 tx=0x7f605c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c042020 con 0x7f60641512d0 2024-03-08T08:29:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c03a040 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60640cca80 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c039070 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.180+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60640cd040 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.181+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60640b84d0 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.182+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f605c005d10 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.182+0000 7f6061ffb640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 0x7f605006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.182+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f605c0ac200 con 0x7f60641512d0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.182+0000 7f606891c640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 0x7f605006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.183+0000 7f606891c640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 0x7f605006d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f606415a2b0 tx=0x7f6058008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.187+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f605c04c070 con 0x7f60641512d0 2024-03-08T08:29:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.407+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f60640749c0 con 0x7f60641512d0 2024-03-08T08:29:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.408+0000 7f6061ffb640 1 -- 172.21.15.12:0/1587455042 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f605c07b080 con 0x7f60641512d0 2024-03-08T08:29:27.409 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 msgr2=0x7f605006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 0x7f605006d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f606415a2b0 tx=0x7f6058008040 comp rx=0 tx=0).stop 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 msgr2=0x7f60640cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f605c00e470 tx=0x7f605c009750 comp rx=0 tx=0).stop 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 shutdown_connections 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f605006b500 0x7f605006d9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60640cab10 0x7f60640caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60640b7b10 0x7f60640b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 --2- 172.21.15.12:0/1587455042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60641512d0 0x7f60640cbd50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 >> 172.21.15.12:0/1587455042 conn(0x7f6064072900 msgr2=0x7f6064160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 shutdown_connections 2024-03-08T08:29:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:27.410+0000 7f606b381640 1 -- 172.21.15.12:0/1587455042 wait complete. 2024-03-08T08:29:27.421 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:29:27.421 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:29:27.421 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-08T08:29:27.421 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-08T08:29:27.421 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:27.422 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:29:27.422 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:29:27.422 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:29:27.422 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-08T08:29:27.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.422 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:27.441 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42601sec preferred_lft 42601sec 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-08T08:29:27.442 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:27.443 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-08T08:29:27.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add name ceph-brx type bridge 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr flush dev ceph-brx 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set ceph-brx up 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-08T08:29:27.443 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:27.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.668 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:29:27.696 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-08T08:29:27.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.697 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:29:27.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.787 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:27.787 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:27.787 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-08T08:29:27.787 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-08T08:29:27.788 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-08T08:29:27.788 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:27.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:27.998 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-08T08:29:28.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.012 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-03-08T08:29:28.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.068 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:28.068 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:28.068 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:28.068 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-08T08:29:28.068 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:28.151 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-08T08:29:28.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-08T08:29:28.152 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:28.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.293 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:28.293 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:28.293 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2024-03-08T08:29:28.293 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.0 master ceph-brx 2024-03-08T08:29:28.293 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:28.336 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-08T08:29:28.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:28.336 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:28.388 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-08T08:29:28.388 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:28.389 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:28.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:29:28.512 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:29:28.567 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:28.568 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:28.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.569 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:29:28.622 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-08T08:29:28.622 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-08T08:29:28.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:29:28.691 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:28.746 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:29:28.746 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:29:28.747 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:29:28.748 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:29:28.749 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:28.749 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:28.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.749 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:29:28.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2024-03-08T08:29:28.752+0000 7f9e88213480 -1 init, newargv = 0x55ce31522570 newargc=15 2024-03-08T08:29:28.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[42497]: starting ceph client 2024-03-08T08:29:28.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[42497]: starting fuse 2024-03-08T08:29:28.767 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-08T08:29:28.767 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:28.767 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-08T08:29:28.768 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-08T08:29:28.769 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:29:28.769 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:28.769 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2024-03-08T08:29:28.769 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:29:28.887 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.42497.asok 2024-03-08T08:29:28.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:29:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:29:28.897 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42497.asok 2024-03-08T08:29:28.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:28.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42497.asok status 2024-03-08T08:29:29.036 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-08T08:29:29.036 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-08T08:29:29.036 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:29:29.036 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:29:29.036 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "42497", 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4606, 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4606 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:29:29.037 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3461462760 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3461462760 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4606 192.168.0.1:0/3461462760", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/3461462760", 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 6, 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 21, 2024-03-08T08:29:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:29:29.039 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-08T08:29:29.039 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-08T08:29:29.039 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-08T08:29:29.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:29:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.158+0000 7f7744607640 1 Processor -- start 2024-03-08T08:29:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.158+0000 7f7744607640 1 -- start start 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f7744607640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f773c0b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f7744607640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c161fb0 con 0x7f773c0d0a30 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f7744607640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c162290 con 0x7f773c161840 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f7744607640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0d1d60 con 0x7f773c15df10 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77423a3640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f773c0d0a30 0x7f773c0d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46276/0 (socket says 172.21.15.12:46276) 2024-03-08T08:29:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77423a3640 1 -- 172.21.15.12:0/2304328685 learned_addr learned my addr 172.21.15.12:0/2304328685 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f7741ba2640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f773c0b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741624520 0 0) 0x7f773c161fb0 con 0x7f773c0d0a30 2024-03-08T08:29:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7728003520 con 0x7f773c0d0a30 2024-03-08T08:29:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815764587 0 0) 0x7f7728003520 con 0x7f773c0d0a30 2024-03-08T08:29:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 msgr2=0x7f773c0b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f773c0b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 >> v1:172.21.15.156:6789/0 conn(0x7f773c161840 legacy=0x7f773c0d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c162570 con 0x7f773c0d0a30 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.159+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f7738001670 con 0x7f773c0d0a30 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 >> v1:172.21.15.12:6789/0 conn(0x7f773c0d0a30 legacy=0x7f773c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77280039e0 0x7f7728003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7728004390 0x7f7728004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f77280050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0d1d60 con 0x7f77280039e0 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772800b9c0 con 0x7f7728004390 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772800bbc0 con 0x7f773c15df10 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77423a3640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77280039e0 0x7f7728003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7728004390 0x7f7728004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7742ba4640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f77280050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 -- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 msgr2=0x7f77280050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c15df10 0x7f77280050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 -- 172.21.15.12:0/2304328685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77280039e0 msgr2=0x7f7728003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77280039e0 0x7f7728003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f7741ba2640 1 -- 172.21.15.12:0/2304328685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728003520 con 0x7f7728004390 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.160+0000 7f77423a3640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77280039e0 0x7f7728003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f7741ba2640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7728004390 0x7f7728004860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f772c009ac0 tx=0x7f7728005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e5da26c918098 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c039070 con 0x7f7728004390 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c02c680 con 0x7f7728004390 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f77413a1640 1 -- 172.21.15.12:0/2304328685 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c009690 con 0x7f7728004390 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f7744607640 1 -- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7728004390 msgr2=0x7f7728004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.161+0000 7f7744607640 1 --2- 172.21.15.12:0/2304328685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7728004390 0x7f7728004860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f772c009ac0 tx=0x7f7728005830 comp rx=0 tx=0).stop 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.162+0000 7f7742ba4640 1 -- 172.21.15.12:0/2304328685 reap_dead start 2024-03-08T08:29:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.162+0000 7f7744607640 1 -- 172.21.15.12:0/2304328685 shutdown_connections 2024-03-08T08:29:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.162+0000 7f7744607640 1 -- 172.21.15.12:0/2304328685 >> 172.21.15.12:0/2304328685 conn(0x7f773c06c710 msgr2=0x7f773c06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.162+0000 7f7744607640 1 -- 172.21.15.12:0/2304328685 shutdown_connections 2024-03-08T08:29:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.162+0000 7f7744607640 1 -- 172.21.15.12:0/2304328685 wait complete. 2024-03-08T08:29:29.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.163+0000 7f7744607640 1 Processor -- start 2024-03-08T08:29:29.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.163+0000 7f7744607640 1 -- start start 2024-03-08T08:29:29.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 0x7f773c147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 0x7f773c0b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c155890 0x7f773c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0b95a0 con 0x7f7728004390 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c161cd0 con 0x7f773c152080 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0b9720 con 0x7f773c155890 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f77423a3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 0x7f773c147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f77423a3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 0x7f773c147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44116/0 (socket says 172.21.15.12:44116) 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 0x7f773c0b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f77423a3640 1 -- 172.21.15.12:0/4164078807 learned_addr learned my addr 172.21.15.12:0/4164078807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7742ba4640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c155890 0x7f773c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 -- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c155890 msgr2=0x7f773c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c155890 0x7f773c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 -- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 msgr2=0x7f773c147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 0x7f773c147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 -- 172.21.15.12:0/4164078807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c148280 con 0x7f773c152080 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7741ba2640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 0x7f773c0b3040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f772c033000 tx=0x7f772c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c039050 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c03a040 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c0b3580 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c003610 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.164+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c154ad0 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.165+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c0749c0 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.166+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f772c047030 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.166+0000 7f7732ffd640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 0x7f772006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.166+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f772c0ab3e0 con 0x7f773c152080 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.166+0000 7f77423a3640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 0x7f772006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.167+0000 7f77423a3640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 0x7f772006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7738001b20 tx=0x7f77380011c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.171+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f772c079c10 con 0x7f773c152080 2024-03-08T08:29:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.389+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f773c0d3540 con 0x7f773c152080 2024-03-08T08:29:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.390+0000 7f7732ffd640 1 -- 172.21.15.12:0/4164078807 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f772c07dcb0 con 0x7f773c152080 2024-03-08T08:29:29.391 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:29:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 msgr2=0x7f772006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 0x7f772006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7738001b20 tx=0x7f77380011c0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 msgr2=0x7f773c0b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 0x7f773c0b3040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f772c033000 tx=0x7f772c004730 comp rx=0 tx=0).stop 2024-03-08T08:29:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 shutdown_connections 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f772006b4d0 0x7f772006d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f773c155890 0x7f773c0cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773c152080 0x7f773c0b3040 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 --2- 172.21.15.12:0/4164078807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7728004390 0x7f773c147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 >> 172.21.15.12:0/4164078807 conn(0x7f773c06c710 msgr2=0x7f773c1528b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 shutdown_connections 2024-03-08T08:29:29.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:29.392+0000 7f7744607640 1 -- 172.21.15.12:0/4164078807 wait complete. 2024-03-08T08:29:29.403 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:29:29.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:29:29.403 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:29:29.404 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-08T08:29:29.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.404 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-08T08:29:29.419 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:29.420 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42599sec preferred_lft 42599sec 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether a2:50:32:30:3f:e5 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::d091:12ff:fe72:2ccf/64 scope link 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether a2:50:32:30:3f:e5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::a050:32ff:fe30:3fe5/64 scope link tentative 2024-03-08T08:29:29.421 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.422 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:29:29.485 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-08T08:29:29.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.487 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:29:29.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.543 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:29.543 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:29.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-08T08:29:29.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-08T08:29:29.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-08T08:29:29.544 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:29.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.639 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-08T08:29:29.653 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-08T08:29:29.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.654 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-03-08T08:29:29.708 INFO:teuthology.orchestra.run.smithi156.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-08T08:29:29.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.708 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:29.709 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:29.709 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:29:29.709 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-08T08:29:29.709 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:29.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.791 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 26:5b:1f:d7:f0:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-08T08:29:29.857 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.858 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::245b:1fff:fed7:f0cf/64 scope link 2024-03-08T08:29:29.858 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.859 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 26:5b:1f:d7:f0:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-08T08:29:29.925 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.926 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::245b:1fff:fed7:f0cf/64 scope link 2024-03-08T08:29:29.926 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:29:29.926 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-08T08:29:29.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-08T08:29:29.927 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:30.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.052 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:29:30.052 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:29:30.052 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.1 up 2024-03-08T08:29:30.052 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.1 master ceph-brx 2024-03-08T08:29:30.052 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:29:30.093 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-08T08:29:30.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:30.093 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:30.147 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-08T08:29:30.147 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:29:30.147 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:30.201 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:29:30.269 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:30.326 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:29:30.326 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:29:30.326 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:29:30.326 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:29:30.326 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:30.327 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:29:30.328 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:29:30.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.329 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:29:30.381 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-08T08:29:30.381 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-08T08:29:30.381 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-08T08:29:30.423 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:29:30.449 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:29:30.505 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:29:30.505 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:29:30.505 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:29:30.505 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:29:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:29:30.507 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:30.508 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:29:30.508 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:29:30.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.508 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:29:30.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:2024-03-08T08:29:30.508+0000 7f3a86183480 -1 init, newargv = 0x55fb0944e340 newargc=15 2024-03-08T08:29:30.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[42950]: starting ceph client 2024-03-08T08:29:30.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[42950]: starting fuse 2024-03-08T08:29:30.523 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-08T08:29:30.523 INFO:teuthology.orchestra.run.smithi156.stdout:44 2024-03-08T08:29:30.523 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-08T08:29:30.523 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:29:30.523 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-08T08:29:30.523 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:30.524 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.3")) 2024-03-08T08:29:30.525 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:29:30.643 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.3.42950.asok 2024-03-08T08:29:30.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:29:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:29:30.654 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.42950.asok 2024-03-08T08:29:30.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.654 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.42950.asok status 2024-03-08T08:29:30.789 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-08T08:29:30.789 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-08T08:29:30.789 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "3", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "42950", 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4618, 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-08T08:29:30.790 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4618 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2073267614 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2073267614 2024-03-08T08:29:30.791 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4618 192.168.0.2:0/2073267614", 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.2:0/2073267614", 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 6, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 21, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-08T08:29:30.792 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-08T08:29:30.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.798 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-03-08T08:29:30.798 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:30.816 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-03-08T08:29:30.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:30.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.817 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:30.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.891 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-03-08T08:29:30.891 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:30.948 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-03-08T08:29:30.948 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:30.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:30.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:31.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:31.021 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-08T08:29:31.021 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:31.036 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-08T08:29:31.036 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:31.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:31.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:31.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:31.104 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-08T08:29:31.104 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:31.159 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-08T08:29:31.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:31.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:31.159 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:31.226 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-08T08:29:31.240 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.350+0000 7f57ea52e640 1 Processor -- start 2024-03-08T08:29:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.350+0000 7f57ea52e640 1 -- start start 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57ea52e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f57e40b9420 0x7f57e415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57ea52e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e415ebc0 con 0x7f57e406c240 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57ea52e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e415eea0 con 0x7f57e40d24b0 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57ea52e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40b6b30 con 0x7f57e40b9420 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57e37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f57e40b9420 0x7f57e415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57e3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f57e406c240 0x7f57e406c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46284/0 (socket says 172.21.15.12:46284) 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.351+0000 7f57e3fff640 1 -- 172.21.15.12:0/360194267 learned_addr learned my addr 172.21.15.12:0/360194267 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.352+0000 7f57e37fe640 1 -- 172.21.15.12:0/360194267 >> v1:172.21.15.156:6789/0 conn(0x7f57e40d24b0 legacy=0x7f57e40d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.352+0000 7f57e37fe640 1 -- 172.21.15.12:0/360194267 >> v1:172.21.15.12:6789/0 conn(0x7f57e406c240 legacy=0x7f57e406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.352+0000 7f57e37fe640 1 -- 172.21.15.12:0/360194267 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40d1cb0 con 0x7f57e40b9420 2024-03-08T08:29:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.352+0000 7f57e2ffd640 1 -- 172.21.15.12:0/360194267 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832443702 0 0) 0x7f57e415ebc0 con 0x7f57e406c240 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.352+0000 7f57e37fe640 1 --2- 172.21.15.12:0/360194267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40b9420 0x7f57e415e220 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57d40097a0 tx=0x7f57d402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea84a300a9c42256 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57e2ffd640 1 -- 172.21.15.12:0/360194267 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4034070 con 0x7f57e40b9420 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57e2ffd640 1 -- 172.21.15.12:0/360194267 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d4030610 con 0x7f57e40b9420 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57e2ffd640 1 -- 172.21.15.12:0/360194267 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d40309e0 con 0x7f57e40b9420 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 -- 172.21.15.12:0/360194267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40b9420 msgr2=0x7f57e415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 --2- 172.21.15.12:0/360194267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40b9420 0x7f57e415e220 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57d40097a0 tx=0x7f57d402b960 comp rx=0 tx=0).stop 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 -- 172.21.15.12:0/360194267 shutdown_connections 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 --2- 172.21.15.12:0/360194267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40b9420 0x7f57e415e220 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 -- 172.21.15.12:0/360194267 >> 172.21.15.12:0/360194267 conn(0x7f57e4072900 msgr2=0x7f57e40d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 -- 172.21.15.12:0/360194267 shutdown_connections 2024-03-08T08:29:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.353+0000 7f57ea52e640 1 -- 172.21.15.12:0/360194267 wait complete. 2024-03-08T08:29:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.354+0000 7f57ea52e640 1 Processor -- start 2024-03-08T08:29:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 -- start start 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e406c240 0x7f57e40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57e40b9420 0x7f57e40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415f180 con 0x7f57e40d24b0 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415bc30 con 0x7f57e406c240 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57ea52e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415e760 con 0x7f57e40b9420 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57e8acb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57e8acb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44124/0 (socket says 172.21.15.12:44124) 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.355+0000 7f57e8acb640 1 -- 172.21.15.12:0/1268871456 learned_addr learned my addr 172.21.15.12:0/1268871456 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e37fe640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57e40b9420 0x7f57e40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e3fff640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e406c240 0x7f57e40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 -- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57e40b9420 msgr2=0x7f57e40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57e40b9420 0x7f57e40cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 -- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e406c240 msgr2=0x7f57e40b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e406c240 0x7f57e40b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 -- 172.21.15.12:0/1268871456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40b59e0 con 0x7f57e40d24b0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e8acb640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00b9a0 tx=0x7f57dc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc019070 con 0x7f57e40d24b0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00cba0 con 0x7f57e40d24b0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e40b5cc0 con 0x7f57e40d24b0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc014440 con 0x7f57e40d24b0 2024-03-08T08:29:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.356+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e40b6280 con 0x7f57e40d24b0 2024-03-08T08:29:31.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.357+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e40b2e90 con 0x7f57e40d24b0 2024-03-08T08:29:31.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.357+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57dc01c070 con 0x7f57e40d24b0 2024-03-08T08:29:31.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.358+0000 7f57e0ff9640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.358+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57dc089d80 con 0x7f57e40d24b0 2024-03-08T08:29:31.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.358+0000 7f57e3fff640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.358+0000 7f57e3fff640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 0x7f57d006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57d8000f60 tx=0x7f57d8007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.362+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f57e40b2e90 con 0x7f57e40d24b0 2024-03-08T08:29:31.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.543+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e40749c0 con 0x7f57e40d24b0 2024-03-08T08:29:31.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.544+0000 7f57e0ff9640 1 -- 172.21.15.12:0/1268871456 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f57dc058c90 con 0x7f57e40d24b0 2024-03-08T08:29:31.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:31.546 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 msgr2=0x7f57d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 0x7f57d006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57d8000f60 tx=0x7f57d8007b50 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 msgr2=0x7f57e40cc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00b9a0 tx=0x7f57dc00be60 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 shutdown_connections 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e40d24b0 0x7f57e40cc520 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57e40b9420 0x7f57e40cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 --2- 172.21.15.12:0/1268871456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e406c240 0x7f57e40b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 >> 172.21.15.12:0/1268871456 conn(0x7f57e4072900 msgr2=0x7f57e415bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 shutdown_connections 2024-03-08T08:29:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.546+0000 7f57ea52e640 1 -- 172.21.15.12:0/1268871456 wait complete. 2024-03-08T08:29:31.558 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:31.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.661+0000 7f044baff640 1 Processor -- start 2024-03-08T08:29:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.661+0000 7f044baff640 1 -- start start 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f044415dcc0 0x7f04440b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f044406c190 0x7f04440b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04440b3f40 0x7f044406abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440ace70 con 0x7f044415dcc0 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0444070670 con 0x7f044406c190 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044baff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044415e0a0 con 0x7f04440b3f40 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044989b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f044415dcc0 0x7f04440b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044989b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f044415dcc0 0x7f04440b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44130/0 (socket says 172.21.15.12:44130) 2024-03-08T08:29:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044989b640 1 -- 172.21.15.12:0/248838258 learned_addr learned my addr 172.21.15.12:0/248838258 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044909a640 1 --2- 172.21.15.12:0/248838258 >> v2:172.21.15.156:3300/0 conn(0x7f044406c190 0x7f04440b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.662+0000 7f044a09c640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04440b3f40 0x7f044406abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 -- 172.21.15.12:0/248838258 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04440b3f40 msgr2=0x7f044406abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04440b3f40 0x7f044406abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 -- 172.21.15.12:0/248838258 >> v2:172.21.15.156:3300/0 conn(0x7f044406c190 msgr2=0x7f04440b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 --2- 172.21.15.12:0/248838258 >> v2:172.21.15.156:3300/0 conn(0x7f044406c190 0x7f04440b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 -- 172.21.15.12:0/248838258 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044406faf0 con 0x7f044415dcc0 2024-03-08T08:29:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f044989b640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044415dcc0 0x7f04440b34c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0444072e40 tx=0x7f043402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20d75cc1b2ed8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.663+0000 7f0448899640 1 -- 172.21.15.12:0/248838258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043402ca70 con 0x7f044415dcc0 2024-03-08T08:29:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f0448899640 1 -- 172.21.15.12:0/248838258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043402cc10 con 0x7f044415dcc0 2024-03-08T08:29:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f0448899640 1 -- 172.21.15.12:0/248838258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04340343f0 con 0x7f044415dcc0 2024-03-08T08:29:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 -- 172.21.15.12:0/248838258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044415dcc0 msgr2=0x7f04440b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044415dcc0 0x7f04440b34c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0444072e40 tx=0x7f043402bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 -- 172.21.15.12:0/248838258 shutdown_connections 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04440b3f40 0x7f044406abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 --2- 172.21.15.12:0/248838258 >> v2:172.21.15.156:3300/0 conn(0x7f044406c190 0x7f04440b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 --2- 172.21.15.12:0/248838258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044415dcc0 0x7f04440b34c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 -- 172.21.15.12:0/248838258 >> 172.21.15.12:0/248838258 conn(0x7f044406db00 msgr2=0x7f044406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 -- 172.21.15.12:0/248838258 shutdown_connections 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.664+0000 7f044baff640 1 -- 172.21.15.12:0/248838258 wait complete. 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.665+0000 7f044baff640 1 Processor -- start 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.665+0000 7f044baff640 1 -- start start 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 0x7f0444142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f044415dcc0 0x7f0444142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0444070390 con 0x7f044406c190 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04440700b0 con 0x7f04440b3f40 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044baff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044406b110 con 0x7f044415dcc0 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44138/0 (socket says 172.21.15.12:44138) 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 -- 172.21.15.12:0/3338236670 learned_addr learned my addr 172.21.15.12:0/3338236670 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 -- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f044415dcc0 msgr2=0x7f0444142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044a09c640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f044415dcc0 0x7f0444142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044909a640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 0x7f0444142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f044415dcc0 0x7f0444142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 -- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 msgr2=0x7f0444142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 0x7f0444142500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044989b640 1 -- 172.21.15.12:0/3338236670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044414fee0 con 0x7f044406c190 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.666+0000 7f044909a640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 0x7f0444142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f044989b640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0434036040 tx=0x7f0434004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434043020 con 0x7f044406c190 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043403e450 con 0x7f044406c190 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044414cca0 con 0x7f044406c190 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434009510 con 0x7f044406c190 2024-03-08T08:29:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.667+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044414d260 con 0x7f044406c190 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.668+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04440afa70 con 0x7f044406c190 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.668+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0434046020 con 0x7f044406c190 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.668+0000 7f04427fc640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.669+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04340ac970 con 0x7f044406c190 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.669+0000 7f044909a640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.669+0000 7f044909a640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 0x7f043006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f044414fc70 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.673+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f043404d030 con 0x7f044406c190 2024-03-08T08:29:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.851+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04440b06b0 con 0x7f044406c190 2024-03-08T08:29:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.851+0000 7f04427fc640 1 -- 172.21.15.12:0/3338236670 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f043407b760 con 0x7f044406c190 2024-03-08T08:29:31.853 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 msgr2=0x7f043006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 0x7f043006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f044414fc70 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 msgr2=0x7f044414e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0434036040 tx=0x7f0434004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 shutdown_connections 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f044415dcc0 0x7f0444142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04440b3f40 0x7f0444142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 --2- 172.21.15.12:0/3338236670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f044406c190 0x7f044414e2a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.853+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 >> 172.21.15.12:0/3338236670 conn(0x7f044406db00 msgr2=0x7f044415b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.854+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 shutdown_connections 2024-03-08T08:29:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.854+0000 7f044baff640 1 -- 172.21.15.12:0/3338236670 wait complete. 2024-03-08T08:29:31.865 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.969+0000 7fbb69f3a640 1 Processor -- start 2024-03-08T08:29:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.969+0000 7fbb69f3a640 1 -- start start 2024-03-08T08:29:31.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb69f3a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb640b6b30 0x7fbb640b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb69f3a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640d1d90 con 0x7fbb64161260 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb69f3a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640d2070 con 0x7fbb6406c160 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb69f3a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640b9ac0 con 0x7fbb640b6b30 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb62ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb640b6b30 0x7fbb640b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb637fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbb64161260 0x7fbb640b93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46296/0 (socket says 172.21.15.12:46296) 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb637fe640 1 -- 172.21.15.12:0/1502811648 learned_addr learned my addr 172.21.15.12:0/1502811648 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb62ffd640 1 -- 172.21.15.12:0/1502811648 >> v1:172.21.15.156:6789/0 conn(0x7fbb6406c160 legacy=0x7fbb640d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb62ffd640 1 -- 172.21.15.12:0/1502811648 >> v1:172.21.15.12:6789/0 conn(0x7fbb64161260 legacy=0x7fbb640b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb62ffd640 1 -- 172.21.15.12:0/1502811648 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640d2350 con 0x7fbb640b6b30 2024-03-08T08:29:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.970+0000 7fbb627fc640 1 -- 172.21.15.12:0/1502811648 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723874986 0 0) 0x7fbb640d1d90 con 0x7fbb64161260 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.971+0000 7fbb62ffd640 1 --2- 172.21.15.12:0/1502811648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb640b6b30 0x7fbb640b6f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb540097a0 tx=0x7fbb5402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ad966e4c2871cb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.971+0000 7fbb627fc640 1 -- 172.21.15.12:0/1502811648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54034070 con 0x7fbb640b6b30 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.971+0000 7fbb627fc640 1 -- 172.21.15.12:0/1502811648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb54030610 con 0x7fbb640b6b30 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.971+0000 7fbb627fc640 1 -- 172.21.15.12:0/1502811648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540309e0 con 0x7fbb640b6b30 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 -- 172.21.15.12:0/1502811648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb640b6b30 msgr2=0x7fbb640b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/1502811648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb640b6b30 0x7fbb640b6f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb540097a0 tx=0x7fbb5402b960 comp rx=0 tx=0).stop 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 -- 172.21.15.12:0/1502811648 shutdown_connections 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/1502811648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb640b6b30 0x7fbb640b6f90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 -- 172.21.15.12:0/1502811648 >> 172.21.15.12:0/1502811648 conn(0x7fbb64072930 msgr2=0x7fbb640d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 -- 172.21.15.12:0/1502811648 shutdown_connections 2024-03-08T08:29:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.972+0000 7fbb69f3a640 1 -- 172.21.15.12:0/1502811648 wait complete. 2024-03-08T08:29:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.973+0000 7fbb69f3a640 1 Processor -- start 2024-03-08T08:29:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.973+0000 7fbb69f3a640 1 -- start start 2024-03-08T08:29:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6406c160 0x7fbb6414dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb640b6b30 0x7fbb641e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640d3540 con 0x7fbb64161260 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640b2ec0 con 0x7fbb6406c160 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb69f3a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640d36c0 con 0x7fbb640b6b30 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44150/0 (socket says 172.21.15.12:44150) 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 -- 172.21.15.12:0/4239730197 learned_addr learned my addr 172.21.15.12:0/4239730197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb62ffd640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb640b6b30 0x7fbb641e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb637fe640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6406c160 0x7fbb6414dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 -- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb640b6b30 msgr2=0x7fbb641e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb640b6b30 0x7fbb641e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 -- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6406c160 msgr2=0x7fbb6414dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6406c160 0x7fbb6414dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 -- 172.21.15.12:0/4239730197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb641e7260 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.974+0000 7fbb63fff640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c7a0 tx=0x7fbb5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.975+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c003b70 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.975+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c003d30 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.975+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb641e7540 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.975+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00aa90 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.975+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641e7b00 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.976+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb640749c0 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.976+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb5c00ac30 con 0x7fbb64161260 2024-03-08T08:29:31.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.976+0000 7fbb5bfff640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 0x7fbb4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:31.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.976+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb5c089fa0 con 0x7fbb64161260 2024-03-08T08:29:31.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.976+0000 7fbb637fe640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 0x7fbb4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:31.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.977+0000 7fbb637fe640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 0x7fbb4406d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c000f60 tx=0x7fbb4c007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:31.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:31.981+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbb5c0587d0 con 0x7fbb64161260 2024-03-08T08:29:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.161+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb640d47f0 con 0x7fbb64161260 2024-03-08T08:29:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.161+0000 7fbb5bfff640 1 -- 172.21.15.12:0/4239730197 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb5c05c870 con 0x7fbb64161260 2024-03-08T08:29:32.163 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.163+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 msgr2=0x7fbb4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.163+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 0x7fbb4406d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c000f60 tx=0x7fbb4c007e40 comp rx=0 tx=0).stop 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.163+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 msgr2=0x7fbb641e87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.163+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c7a0 tx=0x7fbb5c00cc60 comp rx=0 tx=0).stop 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 shutdown_connections 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb4406b500 0x7fbb4406d9c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb64161260 0x7fbb641e87b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb640b6b30 0x7fbb641e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 --2- 172.21.15.12:0/4239730197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6406c160 0x7fbb6414dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 >> 172.21.15.12:0/4239730197 conn(0x7fbb64072930 msgr2=0x7fbb640b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 shutdown_connections 2024-03-08T08:29:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.164+0000 7fbb69f3a640 1 -- 172.21.15.12:0/4239730197 wait complete. 2024-03-08T08:29:32.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.278+0000 7fe4c872c640 1 Processor -- start 2024-03-08T08:29:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.278+0000 7fe4c872c640 1 -- start start 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c872c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4c00b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c872c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00b7250 con 0x7fe4c015dbc0 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c872c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00b7520 con 0x7fe4c00b9760 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c872c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00b8f70 con 0x7fe4c00b85c0 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c64c8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe4c015dbc0 0x7fe4c00b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46302/0 (socket says 172.21.15.12:46302) 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c64c8640 1 -- 172.21.15.12:0/2739656493 learned_addr learned my addr 172.21.15.12:0/2739656493 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c6cc9640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4c00b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.279+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642793975 0 0) 0x7fe4c00b7250 con 0x7fe4c015dbc0 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac003520 con 0x7fe4c015dbc0 2024-03-08T08:29:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243722835 0 0) 0x7fe4ac003520 con 0x7fe4c015dbc0 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 msgr2=0x7fe4c00b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4c00b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 >> v1:172.21.15.156:6789/0 conn(0x7fe4c00b9760 legacy=0x7fe4c00d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c00b7800 con 0x7fe4c015dbc0 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fe4b8001670 con 0x7fe4c015dbc0 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 >> v1:172.21.15.12:6789/0 conn(0x7fe4c015dbc0 legacy=0x7fe4c00b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4ac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac003f40 0x7fe4ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ac004d40 0x7fe4ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00b8f70 con 0x7fe4ac004d40 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac00ba70 con 0x7fe4ac003f40 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac00bc40 con 0x7fe4c00b85c0 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ac004d40 0x7fe4ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c5cc7640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac003f40 0x7fe4ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c64c8640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4ac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 -- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 msgr2=0x7fe4ac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00b85c0 0x7fe4ac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 -- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac003f40 msgr2=0x7fe4ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac003f40 0x7fe4ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.280+0000 7fe4c6cc9640 1 -- 172.21.15.12:0/2739656493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac003520 con 0x7fe4ac004d40 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.281+0000 7fe4c6cc9640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ac004d40 0x7fe4ac0051c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e170 tx=0x7fe4ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab223d4dfd7cb63 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.281+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc039070 con 0x7fe4ac004d40 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.281+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc02cbf0 con 0x7fe4ac004d40 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.281+0000 7fe4c54c6640 1 -- 172.21.15.12:0/2739656493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc009c10 con 0x7fe4ac004d40 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 -- 172.21.15.12:0/2739656493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ac004d40 msgr2=0x7fe4ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 --2- 172.21.15.12:0/2739656493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ac004d40 0x7fe4ac0051c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e170 tx=0x7fe4ac005830 comp rx=0 tx=0).stop 2024-03-08T08:29:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c6cc9640 1 -- 172.21.15.12:0/2739656493 reap_dead start 2024-03-08T08:29:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 -- 172.21.15.12:0/2739656493 shutdown_connections 2024-03-08T08:29:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 -- 172.21.15.12:0/2739656493 >> 172.21.15.12:0/2739656493 conn(0x7fe4c0072720 msgr2=0x7fe4c0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 -- 172.21.15.12:0/2739656493 shutdown_connections 2024-03-08T08:29:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.282+0000 7fe4c872c640 1 -- 172.21.15.12:0/2739656493 wait complete. 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.283+0000 7fe4c872c640 1 Processor -- start 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.283+0000 7fe4c872c640 1 -- start start 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac004d40 0x7fe4c00b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00cbe20 0x7fe4c00cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00b80a0 con 0x7fe4c00d3450 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00b7dc0 con 0x7fe4ac004d40 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00d2220 con 0x7fe4c00cbe20 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44154/0 (socket says 172.21.15.12:44154) 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 -- 172.21.15.12:0/3952905527 learned_addr learned my addr 172.21.15.12:0/3952905527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 -- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00cbe20 msgr2=0x7fe4c00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00cbe20 0x7fe4c00cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 -- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac004d40 msgr2=0x7fe4c00b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac004d40 0x7fe4c00b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 -- 172.21.15.12:0/3952905527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c01f3310 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c5cc7640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000f850 tx=0x7fe4b000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b001a070 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0005320 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c00b2430 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b00072b0 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.284+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c00cce30 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.285+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c00749c0 con 0x7fe4c00d3450 2024-03-08T08:29:32.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.286+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4b001d070 con 0x7fe4c00d3450 2024-03-08T08:29:32.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.286+0000 7fe4b6ffd640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 0x7fe4a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.286+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4b0089a00 con 0x7fe4c00d3450 2024-03-08T08:29:32.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.286+0000 7fe4c64c8640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 0x7fe4a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.287+0000 7fe4c64c8640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 0x7fe4a406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000f60 tx=0x7fe4b8000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.291+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe4c01f3310 con 0x7fe4c00d3450 2024-03-08T08:29:32.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.470+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4c00b37d0 con 0x7fe4c00d3450 2024-03-08T08:29:32.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.470+0000 7fe4b6ffd640 1 -- 172.21.15.12:0/3952905527 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4b0058880 con 0x7fe4c00d3450 2024-03-08T08:29:32.471 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 msgr2=0x7fe4a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 0x7fe4a406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000f60 tx=0x7fe4b8000ed0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 msgr2=0x7fe4c00d38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000f850 tx=0x7fe4b000fd10 comp rx=0 tx=0).stop 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 shutdown_connections 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe4a406b4d0 0x7fe4a406d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4c00cbe20 0x7fe4c00cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c00d3450 0x7fe4c00d38d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 --2- 172.21.15.12:0/3952905527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4ac004d40 0x7fe4c00b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 >> 172.21.15.12:0/3952905527 conn(0x7fe4c0072720 msgr2=0x7fe4c0072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 shutdown_connections 2024-03-08T08:29:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.472+0000 7fe4c872c640 1 -- 172.21.15.12:0/3952905527 wait complete. 2024-03-08T08:29:32.484 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.583+0000 7f3eaf3b6640 1 Processor -- start 2024-03-08T08:29:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.583+0000 7f3eaf3b6640 1 -- start start 2024-03-08T08:29:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eaf3b6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ea815f1a0 0x7f3ea80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eaf3b6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80d0110 con 0x7f3ea80d11b0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eaf3b6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80d03f0 con 0x7f3ea809a310 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eaf3b6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea815f580 con 0x7f3ea815f1a0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eac951640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ea80d11b0 0x7f3ea8099c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46306/0 (socket says 172.21.15.12:46306) 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.584+0000 7f3eac951640 1 -- 172.21.15.12:0/7011014 learned_addr learned my addr 172.21.15.12:0/7011014 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead152640 1 --2- 172.21.15.12:0/7011014 >> v2:172.21.15.156:3300/0 conn(0x7f3ea815f1a0 0x7f3ea80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150398044 0 0) 0x7f3ea80d0110 con 0x7f3ea80d11b0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e94003520 con 0x7f3ea80d11b0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721475215 0 0) 0x7f3ea80d03f0 con 0x7f3ea809a310 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea80d0110 con 0x7f3ea809a310 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 560083631 0 0) 0x7f3e94003520 con 0x7f3ea80d11b0 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 >> v2:172.21.15.156:3300/0 conn(0x7f3ea815f1a0 msgr2=0x7f3ea80cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 --2- 172.21.15.12:0/7011014 >> v2:172.21.15.156:3300/0 conn(0x7f3ea815f1a0 0x7f3ea80cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 >> v1:172.21.15.156:6789/0 conn(0x7f3ea809a310 legacy=0x7f3ea80cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80d06d0 con 0x7f3ea80d11b0 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3e98001670 con 0x7f3ea80d11b0 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 >> v1:172.21.15.12:6789/0 conn(0x7f3ea80d11b0 legacy=0x7f3ea8099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e94003b40 0x7f3e94003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e94004800 0x7f3e94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 0x7f3e9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9400c010 con 0x7f3e940051c0 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9400c1c0 con 0x7f3e94004800 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9400c3c0 con 0x7f3e94003b40 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 0x7f3e9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3eac951640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e94004800 0x7f3e94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead152640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e94003b40 0x7f3e94003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 -- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e94003b40 msgr2=0x7f3e94003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e94003b40 0x7f3e94003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 -- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e94004800 msgr2=0x7f3e94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e94004800 0x7f3e94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.585+0000 7f3ead953640 1 -- 172.21.15.12:0/7011014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94003520 con 0x7f3e940051c0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.586+0000 7f3ead953640 1 -- 172.21.15.12:0/7011014 reap_dead start 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.586+0000 7f3ead953640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 0x7f3e9400b9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0001fd0 tx=0x7f3e94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f4579df7787b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.586+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0001bf0 con 0x7f3e940051c0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.586+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0003b70 con 0x7f3e940051c0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.586+0000 7f3e9ffff640 1 -- 172.21.15.12:0/7011014 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00050d0 con 0x7f3e940051c0 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 msgr2=0x7f3e9400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 0x7f3e9400b9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0001fd0 tx=0x7f3e94005830 comp rx=0 tx=0).stop 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/7011014 shutdown_connections 2024-03-08T08:29:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/7011014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e940051c0 0x7f3e9400b9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/7011014 >> 172.21.15.12:0/7011014 conn(0x7f3ea80b3490 msgr2=0x7f3ea80b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/7011014 shutdown_connections 2024-03-08T08:29:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.587+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/7011014 wait complete. 2024-03-08T08:29:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.588+0000 7f3eaf3b6640 1 Processor -- start 2024-03-08T08:29:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.588+0000 7f3eaf3b6640 1 -- start start 2024-03-08T08:29:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e940051c0 0x7f3ea809f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea815ba40 0x7f3ea80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806bf00 con 0x7f3ea814e5b0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80d0c90 con 0x7f3ea815ba40 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80cfb10 con 0x7f3e940051c0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44164/0 (socket says 172.21.15.12:44164) 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 -- 172.21.15.12:0/3633404878 learned_addr learned my addr 172.21.15.12:0/3633404878 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 -- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e940051c0 msgr2=0x7f3ea809f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e940051c0 0x7f3ea809f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 -- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea815ba40 msgr2=0x7f3ea80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea815ba40 0x7f3ea80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 -- 172.21.15.12:0/3633404878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80ca710 con 0x7f3ea814e5b0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eac951640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3e98007340 tx=0x7f3e98007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9800f8e0 con 0x7f3ea814e5b0 2024-03-08T08:29:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98018070 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9800fd00 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.589+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea814bac0 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.590+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea80c9ed0 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.591+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea815be90 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.591+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e9801d460 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.591+0000 7f3e9dffb640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.591+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e9808a1f0 con 0x7f3ea814e5b0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.591+0000 7f3ead152640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.592+0000 7f3ead152640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009a40 tx=0x7f3ea4009320 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.596+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ea815be90 con 0x7f3ea814e5b0 2024-03-08T08:29:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.780+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ea80749c0 con 0x7f3ea814e5b0 2024-03-08T08:29:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.781+0000 7f3e9dffb640 1 -- 172.21.15.12:0/3633404878 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f3e98059100 con 0x7f3ea814e5b0 2024-03-08T08:29:32.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:32.783 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 msgr2=0x7f3e8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009a40 tx=0x7f3ea4009320 comp rx=0 tx=0).stop 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 msgr2=0x7f3ea815b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3e98007340 tx=0x7f3e98007800 comp rx=0 tx=0).stop 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 shutdown_connections 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea815ba40 0x7f3ea80c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ea814e5b0 0x7f3ea815b500 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 --2- 172.21.15.12:0/3633404878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e940051c0 0x7f3ea809f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 >> 172.21.15.12:0/3633404878 conn(0x7f3ea80b3490 msgr2=0x7f3ea80b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 shutdown_connections 2024-03-08T08:29:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.783+0000 7f3eaf3b6640 1 -- 172.21.15.12:0/3633404878 wait complete. 2024-03-08T08:29:32.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:32.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:32.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.899+0000 7f3324a71640 1 Processor -- start 2024-03-08T08:29:32.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.899+0000 7f3324a71640 1 -- start start 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f3324a71640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f332015ddc0 0x7f33200d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f3324a71640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332015e1a0 con 0x7f33200b7bf0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f3324a71640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33200b8790 con 0x7f33200d0f40 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f3324a71640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33200b9030 con 0x7f332015ddc0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331ed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f33200b7bf0 0x7f33200d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46316/0 (socket says 172.21.15.12:46316) 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331ed9d640 1 -- 172.21.15.12:0/1701766138 learned_addr learned my addr 172.21.15.12:0/1701766138 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331e59c640 1 --2- 172.21.15.12:0/1701766138 >> v2:172.21.15.156:3300/0 conn(0x7f332015ddc0 0x7f33200d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337627846 0 0) 0x7f332015e1a0 con 0x7f33200b7bf0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f330c003520 con 0x7f33200b7bf0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181236734 0 0) 0x7f330c003520 con 0x7f33200b7bf0 2024-03-08T08:29:32.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 >> v2:172.21.15.156:3300/0 conn(0x7f332015ddc0 msgr2=0x7f33200d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 --2- 172.21.15.12:0/1701766138 >> v2:172.21.15.156:3300/0 conn(0x7f332015ddc0 0x7f33200d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 >> v1:172.21.15.156:6789/0 conn(0x7f33200d0f40 legacy=0x7f33200d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200b8a70 con 0x7f33200b7bf0 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f33180019f0 con 0x7f33200b7bf0 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 >> v1:172.21.15.12:6789/0 conn(0x7f33200b7bf0 legacy=0x7f33200d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 0x7f330c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c004800 0x7f330c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.900+0000 7f331dd9b640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f330c0051c0 0x7f330c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c00c030 con 0x7f330c0039b0 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c00c1e0 con 0x7f330c0051c0 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c00c3e0 con 0x7f330c004800 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 0x7f330c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331ed9d640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f330c0051c0 0x7f330c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f3317fff640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c004800 0x7f330c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 -- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c004800 msgr2=0x7f330c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c004800 0x7f330c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 -- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f330c0051c0 msgr2=0x7f330c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f330c0051c0 0x7f330c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331e59c640 1 -- 172.21.15.12:0/1701766138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c003520 con 0x7f330c0039b0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331ed9d640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f330c0051c0 0x7f330c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.901+0000 7f331ed9d640 1 -- 172.21.15.12:0/1701766138 reap_dead start 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f331e59c640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 0x7f330c003dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3310009ac0 tx=0x7f330c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7def8615f8db8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310039070 con 0x7f330c0039b0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331002c6b0 con 0x7f330c0039b0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f331dd9b640 1 -- 172.21.15.12:0/1701766138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310009690 con 0x7f330c0039b0 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 -- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 msgr2=0x7f330c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 0x7f330c003dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3310009ac0 tx=0x7f330c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 -- 172.21.15.12:0/1701766138 shutdown_connections 2024-03-08T08:29:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 --2- 172.21.15.12:0/1701766138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f330c0039b0 0x7f330c003dd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 -- 172.21.15.12:0/1701766138 >> 172.21.15.12:0/1701766138 conn(0x7f3320072720 msgr2=0x7f3320072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 -- 172.21.15.12:0/1701766138 shutdown_connections 2024-03-08T08:29:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.902+0000 7f3324a71640 1 -- 172.21.15.12:0/1701766138 wait complete. 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 Processor -- start 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 -- start start 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c0039b0 0x7f3320161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 0x7f3320146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200b9980 con 0x7f33201468c0 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200d1d20 con 0x7f3320145bd0 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.904+0000 7f3324a71640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200d0a00 con 0x7f330c0039b0 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44176/0 (socket says 172.21.15.12:44176) 2024-03-08T08:29:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 -- 172.21.15.12:0/3634463260 learned_addr learned my addr 172.21.15.12:0/3634463260 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f331e59c640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c0039b0 0x7f3320161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f331ed9d640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 0x7f3320146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 -- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c0039b0 msgr2=0x7f3320161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c0039b0 0x7f3320161060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 -- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 msgr2=0x7f3320146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 0x7f3320146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 -- 172.21.15.12:0/3634463260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200cc710 con 0x7f33201468c0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f331ed9d640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 0x7f3320146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3317fff640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f330800e500 tx=0x7f330800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080053a0 con 0x7f33201468c0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308018070 con 0x7f33201468c0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308003810 con 0x7f33201468c0 2024-03-08T08:29:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3320153c50 con 0x7f33201468c0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.905+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33201615a0 con 0x7f33201468c0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.906+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33200749c0 con 0x7f33201468c0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.906+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33080039b0 con 0x7f33201468c0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.907+0000 7f3316ffd640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 0x7f330006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.907+0000 7f331e59c640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 0x7f330006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.907+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3308089f70 con 0x7f33201468c0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.907+0000 7f331e59c640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 0x7f330006d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3310036000 tx=0x7f3310030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:32.911+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3308058d70 con 0x7f33201468c0 2024-03-08T08:29:33.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.092+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33200d3fa0 con 0x7f33201468c0 2024-03-08T08:29:33.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.093+0000 7f3316ffd640 1 -- 172.21.15.12:0/3634463260 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3308058670 con 0x7f33201468c0 2024-03-08T08:29:33.094 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:33.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 msgr2=0x7f330006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 0x7f330006d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3310036000 tx=0x7f3310030000 comp rx=0 tx=0).stop 2024-03-08T08:29:33.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 msgr2=0x7f33200cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f330800e500 tx=0x7f330800e9c0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 shutdown_connections 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f330006b500 0x7f330006d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3320145bd0 0x7f3320146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33201468c0 0x7f33200cced0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 --2- 172.21.15.12:0/3634463260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330c0039b0 0x7f3320161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 >> 172.21.15.12:0/3634463260 conn(0x7f3320072720 msgr2=0x7f33200b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 shutdown_connections 2024-03-08T08:29:33.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.095+0000 7f3324a71640 1 -- 172.21.15.12:0/3634463260 wait complete. 2024-03-08T08:29:33.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:33.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 Processor -- start 2024-03-08T08:29:33.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 -- start start 2024-03-08T08:29:33.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbca415dec0 0x7fbca4161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbca4161ed0 0x7fbca40d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca40d2140 con 0x7fbca415dec0 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca40b6be0 con 0x7fbca4161ed0 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.210+0000 7fbca9476640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca415e2a0 con 0x7fbca40d0790 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbca415dec0 0x7fbca4161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbca415dec0 0x7fbca4161990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44190/0 (socket says 172.21.15.12:44190) 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 -- 172.21.15.12:0/3636986833 learned_addr learned my addr 172.21.15.12:0/3636986833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca27fc640 1 --2- 172.21.15.12:0/3636986833 >> v2:172.21.15.156:3300/0 conn(0x7fbca4161ed0 0x7fbca40d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca37fe640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 -- 172.21.15.12:0/3636986833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 msgr2=0x7fbca40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 -- 172.21.15.12:0/3636986833 >> v2:172.21.15.156:3300/0 conn(0x7fbca4161ed0 msgr2=0x7fbca40d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/3636986833 >> v2:172.21.15.156:3300/0 conn(0x7fbca4161ed0 0x7fbca40d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca2ffd640 1 -- 172.21.15.12:0/3636986833 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4162350 con 0x7fbca415dec0 2024-03-08T08:29:33.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.211+0000 7fbca37fe640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:33.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca4161990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbca408ca20 tx=0x7fbc9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23158b5eaa8c4c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca1ffb640 1 -- 172.21.15.12:0/3636986833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94035020 con 0x7fbca415dec0 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca1ffb640 1 -- 172.21.15.12:0/3636986833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94031870 con 0x7fbca415dec0 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca1ffb640 1 -- 172.21.15.12:0/3636986833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94031c40 con 0x7fbca415dec0 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 -- 172.21.15.12:0/3636986833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 msgr2=0x7fbca4161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca4161990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbca408ca20 tx=0x7fbc9402b910 comp rx=0 tx=0).stop 2024-03-08T08:29:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 -- 172.21.15.12:0/3636986833 shutdown_connections 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 --2- 172.21.15.12:0/3636986833 >> v2:172.21.15.156:3300/0 conn(0x7fbca4161ed0 0x7fbca40d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 --2- 172.21.15.12:0/3636986833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca4161990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 -- 172.21.15.12:0/3636986833 >> 172.21.15.12:0/3636986833 conn(0x7fbca406c240 msgr2=0x7fbca406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.212+0000 7fbca9476640 1 -- 172.21.15.12:0/3636986833 shutdown_connections 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.213+0000 7fbca9476640 1 -- 172.21.15.12:0/3636986833 wait complete. 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 Processor -- start 2024-03-08T08:29:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 -- start start 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4161ed0 0x7fbca4160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40d0bd0 con 0x7fbca415dec0 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40d1dd0 con 0x7fbca4161ed0 2024-03-08T08:29:33.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.214+0000 7fbca9476640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40b94f0 con 0x7fbca40d0790 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44192/0 (socket says 172.21.15.12:44192) 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 -- 172.21.15.12:0/1504945041 learned_addr learned my addr 172.21.15.12:0/1504945041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca37fe640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4161ed0 0x7fbca4160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 -- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 msgr2=0x7fbca40c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 -- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4161ed0 msgr2=0x7fbca4160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4161ed0 0x7fbca4160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 -- 172.21.15.12:0/1504945041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca415e800 con 0x7fbca415dec0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca27fc640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800e9b0 tx=0x7fbc9800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98017070 con 0x7fbca415dec0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9800c9a0 con 0x7fbca415dec0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca40c7ad0 con 0x7fbca415dec0 2024-03-08T08:29:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9800cd90 con 0x7fbca415dec0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.215+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca40c8090 con 0x7fbca415dec0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.216+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca40749c0 con 0x7fbca415dec0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.217+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc9801a070 con 0x7fbca415dec0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.217+0000 7fbc8b7fe640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 0x7fbc9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.217+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc9808a160 con 0x7fbca415dec0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.217+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 0x7fbc9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.218+0000 7fbca2ffd640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 0x7fbc9006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036000 tx=0x7fbc9403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.221+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbc98059070 con 0x7fbca415dec0 2024-03-08T08:29:33.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.404+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbca40d1750 con 0x7fbca415dec0 2024-03-08T08:29:33.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.404+0000 7fbc8b7fe640 1 -- 172.21.15.12:0/1504945041 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc98058970 con 0x7fbca415dec0 2024-03-08T08:29:33.406 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 msgr2=0x7fbc9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 0x7fbc9006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036000 tx=0x7fbc9403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 msgr2=0x7fbca415f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800e9b0 tx=0x7fbc9800ee70 comp rx=0 tx=0).stop 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 shutdown_connections 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9006b4d0 0x7fbc9006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4161ed0 0x7fbca4160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca415dec0 0x7fbca415f420 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 --2- 172.21.15.12:0/1504945041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40d0790 0x7fbca40c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 >> 172.21.15.12:0/1504945041 conn(0x7fbca406c240 msgr2=0x7fbca415c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 shutdown_connections 2024-03-08T08:29:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.407+0000 7fbca9476640 1 -- 172.21.15.12:0/1504945041 wait complete. 2024-03-08T08:29:33.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:33.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.522+0000 7f451c35e640 1 Processor -- start 2024-03-08T08:29:33.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.522+0000 7f451c35e640 1 -- start start 2024-03-08T08:29:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451c35e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f45140d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451c35e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45140d2050 0x7f45140b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451c35e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140b7250 con 0x7f451415dbc0 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451c35e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140b7520 con 0x7f45140d2050 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451c35e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b7800 con 0x7f45140b9760 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451a0fa640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f451415dbc0 0x7f45140b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46332/0 (socket says 172.21.15.12:46332) 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451a0fa640 1 -- 172.21.15.12:0/3473302925 learned_addr learned my addr 172.21.15.12:0/3473302925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293258310 0 0) 0x7f45140b7250 con 0x7f451415dbc0 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003520 con 0x7f451415dbc0 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f451a8fb640 1 --2- 172.21.15.12:0/3473302925 >> v2:172.21.15.156:3300/0 conn(0x7f45140d2050 0x7f45140b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45198f9640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f45140d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 307932975 0 0) 0x7f4500003520 con 0x7f451415dbc0 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 msgr2=0x7f45140d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f45140d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 >> v2:172.21.15.156:3300/0 conn(0x7f45140d2050 msgr2=0x7f45140b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 --2- 172.21.15.12:0/3473302925 >> v2:172.21.15.156:3300/0 conn(0x7f45140d2050 0x7f45140b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140b7ae0 con 0x7f451415dbc0 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.523+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f450c001670 con 0x7f451415dbc0 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 >> v1:172.21.15.12:6789/0 conn(0x7f451415dbc0 legacy=0x7f45140b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f4500003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f4500004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4500004d40 0x7f45000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b7520 con 0x7f45000039b0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b7800 con 0x7f4500004d40 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450000bb50 con 0x7f45140b9760 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f4500003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a8fb640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4500004d40 0x7f45000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f45198f9640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f4500004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 -- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 msgr2=0x7f4500004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140b9760 0x7f4500004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 -- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4500004d40 msgr2=0x7f45000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4500004d40 0x7f45000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a0fa640 1 -- 172.21.15.12:0/3473302925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4500003520 con 0x7f45000039b0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.524+0000 7f451a8fb640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4500004d40 0x7f45000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451a0fa640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f4500003dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c240 tx=0x7f4500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5dc4cd44d32b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c007d10 con 0x7f45000039b0 2024-03-08T08:29:33.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c030070 con 0x7f45000039b0 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f45190f8640 1 -- 172.21.15.12:0/3473302925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c02cd80 con 0x7f45000039b0 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 -- 172.21.15.12:0/3473302925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 msgr2=0x7f4500003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 --2- 172.21.15.12:0/3473302925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f4500003dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c240 tx=0x7f4500005830 comp rx=0 tx=0).stop 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451a8fb640 1 -- 172.21.15.12:0/3473302925 reap_dead start 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 -- 172.21.15.12:0/3473302925 shutdown_connections 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 -- 172.21.15.12:0/3473302925 >> 172.21.15.12:0/3473302925 conn(0x7f4514072720 msgr2=0x7f4514072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 -- 172.21.15.12:0/3473302925 shutdown_connections 2024-03-08T08:29:33.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.525+0000 7f451c35e640 1 -- 172.21.15.12:0/3473302925 wait complete. 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 Processor -- start 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 -- start start 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140ad810 0x7f4514157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45140b4540 0x7f45140b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451415dfa0 con 0x7f45000039b0 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b80a0 con 0x7f45140b4540 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451c35e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140d1b10 con 0x7f45140ad810 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451a0fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451a0fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44200/0 (socket says 172.21.15.12:44200) 2024-03-08T08:29:33.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.527+0000 7f451a0fa640 1 -- 172.21.15.12:0/315898762 learned_addr learned my addr 172.21.15.12:0/315898762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f45198f9640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140ad810 0x7f4514157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a8fb640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45140b4540 0x7f45140b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 -- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140ad810 msgr2=0x7f4514157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140ad810 0x7f4514157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 -- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45140b4540 msgr2=0x7f45140b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45140b4540 0x7f45140b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 -- 172.21.15.12:0/315898762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140ab960 con 0x7f45000039b0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451a0fa640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c360 tx=0x7f45140b7cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c045070 con 0x7f45000039b0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c004bf0 con 0x7f45000039b0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45140abc40 con 0x7f45000039b0 2024-03-08T08:29:33.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c007d10 con 0x7f45000039b0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.528+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140ac200 con 0x7f45000039b0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.529+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140749c0 con 0x7f45000039b0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.529+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f450c007070 con 0x7f45000039b0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.530+0000 7f450affd640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 0x7f44f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.530+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f450c0ab260 con 0x7f45000039b0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.530+0000 7f45198f9640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 0x7f44f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.530+0000 7f45198f9640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 0x7f44f806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.534+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f450c07a150 con 0x7f45000039b0 2024-03-08T08:29:33.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.714+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45140d4580 con 0x7f45000039b0 2024-03-08T08:29:33.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.714+0000 7f450affd640 1 -- 172.21.15.12:0/315898762 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f450c079a50 con 0x7f45000039b0 2024-03-08T08:29:33.715 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 msgr2=0x7f44f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 0x7f44f806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504008040 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 msgr2=0x7f45140ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c360 tx=0x7f45140b7cb0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 shutdown_connections 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f44f806b4d0 0x7f44f806d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45140b4540 0x7f45140b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45140ad810 0x7f4514157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 --2- 172.21.15.12:0/315898762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45000039b0 0x7f45140ab280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 >> 172.21.15.12:0/315898762 conn(0x7f4514072720 msgr2=0x7f45140d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 shutdown_connections 2024-03-08T08:29:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.716+0000 7f451c35e640 1 -- 172.21.15.12:0/315898762 wait complete. 2024-03-08T08:29:33.728 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:33.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.828+0000 7f82795b5640 1 Processor -- start 2024-03-08T08:29:33.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.828+0000 7f82795b5640 1 -- start start 2024-03-08T08:29:33.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82795b5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f827415aa70 0x7f82740c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82795b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f82740d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82795b5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740cf870 con 0x7f82740d0b60 2024-03-08T08:29:33.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82795b5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827415ae50 con 0x7f827415aa70 2024-03-08T08:29:33.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82795b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cfb10 con 0x7f82740cf410 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82737fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f82740d0b60 0x7f82740cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46348/0 (socket says 172.21.15.12:46348) 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82737fe640 1 -- 172.21.15.12:0/4248689724 learned_addr learned my addr 172.21.15.12:0/4248689724 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f82727fc640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f82740d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f8272ffd640 1 --2- 172.21.15.12:0/4248689724 >> v2:172.21.15.156:3300/0 conn(0x7f827415aa70 0x7f82740c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696865648 0 0) 0x7f82740cf870 con 0x7f82740d0b60 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.829+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8260003520 con 0x7f82740d0b60 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480294426 0 0) 0x7f8260003520 con 0x7f82740d0b60 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 msgr2=0x7f82740d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f82740d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 >> v2:172.21.15.156:3300/0 conn(0x7f827415aa70 msgr2=0x7f82740c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 --2- 172.21.15.12:0/4248689724 >> v2:172.21.15.156:3300/0 conn(0x7f827415aa70 0x7f82740c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740cfdf0 con 0x7f82740d0b60 2024-03-08T08:29:33.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f82640019f0 con 0x7f82740d0b60 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 >> v1:172.21.15.12:6789/0 conn(0x7f82740d0b60 legacy=0x7f82740cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82600039b0 0x7f8260003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f8260004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8260004d40 0x7f82600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cfb10 con 0x7f8260004d40 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826000b9c0 con 0x7f82600039b0 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826000bb40 con 0x7f82740cf410 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8260004d40 0x7f82600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f8272ffd640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82600039b0 0x7f8260003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82727fc640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f8260004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 -- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 msgr2=0x7f8260004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f8260004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 -- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82600039b0 msgr2=0x7f8260003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:33.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82600039b0 0x7f8260003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82737fe640 1 -- 172.21.15.12:0/4248689724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260003520 con 0x7f8260004d40 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.830+0000 7f82727fc640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82740cf410 0x7f8260004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.831+0000 7f82737fe640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8260004d40 0x7f82600051c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f826400cb30 tx=0x7f8260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b41d3e9909d8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.831+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400c780 con 0x7f8260004d40 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.831+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264033070 con 0x7f8260004d40 2024-03-08T08:29:33.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.831+0000 7f8271ffb640 1 -- 172.21.15.12:0/4248689724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826402fc40 con 0x7f8260004d40 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 -- 172.21.15.12:0/4248689724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8260004d40 msgr2=0x7f82600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 --2- 172.21.15.12:0/4248689724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8260004d40 0x7f82600051c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f826400cb30 tx=0x7f8260005830 comp rx=0 tx=0).stop 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82737fe640 1 -- 172.21.15.12:0/4248689724 reap_dead start 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 -- 172.21.15.12:0/4248689724 shutdown_connections 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 -- 172.21.15.12:0/4248689724 >> 172.21.15.12:0/4248689724 conn(0x7f8274099a40 msgr2=0x7f8274099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 -- 172.21.15.12:0/4248689724 shutdown_connections 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.832+0000 7f82795b5640 1 -- 172.21.15.12:0/4248689724 wait complete. 2024-03-08T08:29:33.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.833+0000 7f82795b5640 1 Processor -- start 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.833+0000 7f82795b5640 1 -- start start 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8274153540 0x7f82741546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8274154200 0x7f8274156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cc590 con 0x7f8274156bc0 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cb060 con 0x7f8274153540 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82795b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cc710 con 0x7f8274154200 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44214/0 (socket says 172.21.15.12:44214) 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 -- 172.21.15.12:0/3217120027 learned_addr learned my addr 172.21.15.12:0/3217120027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f8272ffd640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8274153540 0x7f82741546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82727fc640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8274154200 0x7f8274156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 -- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8274154200 msgr2=0x7f8274156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8274154200 0x7f8274156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 -- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8274153540 msgr2=0x7f82741546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:33.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8274153540 0x7f82741546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 -- 172.21.15.12:0/3217120027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274155050 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f82737fe640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f826400c2c0 tx=0x7f8264001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264002040 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.834+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826400c760 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.835+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82740afff0 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.835+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264038690 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.835+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82740b05b0 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.835+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82740749c0 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.836+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8264038830 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.836+0000 7f826b7fe640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 0x7f825406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.836+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82640ae450 con 0x7f8274156bc0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.836+0000 7f8272ffd640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 0x7f825406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:33.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.837+0000 7f8272ffd640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 0x7f825406d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f826c009bc0 tx=0x7f826c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:33.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:33.841+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f826404a060 con 0x7f8274156bc0 2024-03-08T08:29:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.022+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8274157050 con 0x7f8274156bc0 2024-03-08T08:29:34.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.023+0000 7f826b7fe640 1 -- 172.21.15.12:0/3217120027 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f826407d390 con 0x7f8274156bc0 2024-03-08T08:29:34.025 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:34.025 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 msgr2=0x7f825406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 0x7f825406d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f826c009bc0 tx=0x7f826c009190 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 msgr2=0x7f82741ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f826400c2c0 tx=0x7f8264001dc0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 shutdown_connections 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f825406b500 0x7f825406d9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8274156bc0 0x7f82741ef9b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8274154200 0x7f8274156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 --2- 172.21.15.12:0/3217120027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8274153540 0x7f82741546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 >> 172.21.15.12:0/3217120027 conn(0x7f8274099a40 msgr2=0x7f827406abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 shutdown_connections 2024-03-08T08:29:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.025+0000 7f82795b5640 1 -- 172.21.15.12:0/3217120027 wait complete. 2024-03-08T08:29:34.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:34.038 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:34.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.141+0000 7f7e30c47640 1 Processor -- start 2024-03-08T08:29:34.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.141+0000 7f7e30c47640 1 -- start start 2024-03-08T08:29:34.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e30c47640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e2c15ddc0 0x7f7e2c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e30c47640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c15e1a0 con 0x7f7e2c0b7bf0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e30c47640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0b8790 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e30c47640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b9030 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2ad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7e2c0b7bf0 0x7f7e2c0d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46360/0 (socket says 172.21.15.12:46360) 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2ad9d640 1 -- 172.21.15.12:0/4099010903 learned_addr learned my addr 172.21.15.12:0/4099010903 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2a59c640 1 --2- 172.21.15.12:0/4099010903 >> v2:172.21.15.156:3300/0 conn(0x7f7e2c15ddc0 0x7f7e2c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454568807 0 0) 0x7f7e2c15e1a0 con 0x7f7e2c0b7bf0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18003520 con 0x7f7e2c0b7bf0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530935435 0 0) 0x7f7e18003520 con 0x7f7e2c0b7bf0 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.142+0000 7f7e2a59c640 1 -- 172.21.15.12:0/4099010903 >> v1:172.21.15.156:6789/0 conn(0x7f7e2c0d0f40 legacy=0x7f7e2c0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.143+0000 7f7e2a59c640 1 -- 172.21.15.12:0/4099010903 >> v1:172.21.15.12:6789/0 conn(0x7f7e2c0b7bf0 legacy=0x7f7e2c0d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.143+0000 7f7e2a59c640 1 -- 172.21.15.12:0/4099010903 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0b8a70 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.143+0000 7f7e2a59c640 1 --2- 172.21.15.12:0/4099010903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c0d0a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90a14905d6ff8da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.143+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24038070 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24031500 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e2959a640 1 -- 172.21.15.12:0/4099010903 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0019f0 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 -- 172.21.15.12:0/4099010903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 msgr2=0x7f7e2c0d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 --2- 172.21.15.12:0/4099010903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c0d0a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402b820 comp rx=0 tx=0).stop 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 -- 172.21.15.12:0/4099010903 shutdown_connections 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 --2- 172.21.15.12:0/4099010903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c0d0a00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 -- 172.21.15.12:0/4099010903 >> 172.21.15.12:0/4099010903 conn(0x7f7e2c072720 msgr2=0x7f7e2c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 -- 172.21.15.12:0/4099010903 shutdown_connections 2024-03-08T08:29:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.144+0000 7f7e30c47640 1 -- 172.21.15.12:0/4099010903 wait complete. 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.145+0000 7f7e30c47640 1 Processor -- start 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.145+0000 7f7e30c47640 1 -- start start 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0b7bf0 0x7f7e2c160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c1f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b9980 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0d1d20 con 0x7f7e2c15ddc0 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e30c47640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0d2fd0 con 0x7f7e2c0b7bf0 2024-03-08T08:29:34.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e29d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e29d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44224/0 (socket says 172.21.15.12:44224) 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e29d9b640 1 -- 172.21.15.12:0/2093807785 learned_addr learned my addr 172.21.15.12:0/2093807785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e2a59c640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0b7bf0 0x7f7e2c160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.146+0000 7f7e2ad9d640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c1f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 -- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0b7bf0 msgr2=0x7f7e2c160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0b7bf0 0x7f7e2c160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 -- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 msgr2=0x7f7e2c1f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c1f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 -- 172.21.15.12:0/2093807785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c14d260 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e29d9b640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c450 tx=0x7f7e1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e140052d0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14005470 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c157c10 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14012870 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.147+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c1581d0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.148+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0749c0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.148+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e14012a10 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.149+0000 7f7e22ffd640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.149+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e14089ed0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.149+0000 7f7e2a59c640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.149+0000 7f7e2a59c640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7e24035000 tx=0x7f7e24008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.153+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e14058cc0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.333+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e2c0d34e0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.334+0000 7f7e22ffd640 1 -- 172.21.15.12:0/2093807785 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e140585c0 con 0x7f7e2c0d0f40 2024-03-08T08:29:34.335 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:34.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 msgr2=0x7f7e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7e24035000 tx=0x7f7e24008030 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 msgr2=0x7f7e2c14cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c450 tx=0x7f7e1400c910 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 shutdown_connections 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c15ddc0 0x7f7e2c1f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2c0d0f40 0x7f7e2c14cd20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 --2- 172.21.15.12:0/2093807785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0b7bf0 0x7f7e2c160530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 >> 172.21.15.12:0/2093807785 conn(0x7f7e2c072720 msgr2=0x7f7e2c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 shutdown_connections 2024-03-08T08:29:34.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.336+0000 7f7e30c47640 1 -- 172.21.15.12:0/2093807785 wait complete. 2024-03-08T08:29:34.348 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:34.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.450+0000 7f4981895640 1 Processor -- start 2024-03-08T08:29:34.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.450+0000 7f4981895640 1 -- start start 2024-03-08T08:29:34.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4981895640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 0x7f497c0cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4981895640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f497c0cffb0 0x7f497c15c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4981895640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c15f710 con 0x7f497c1601d0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4981895640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c0d0400 con 0x7f497c0cffb0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4981895640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15cd40 con 0x7f497c15b950 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f497c0cffb0 0x7f497c15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497a7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f497c1601d0 0x7f497c15bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46362/0 (socket says 172.21.15.12:46362) 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497a7fc640 1 -- 172.21.15.12:0/171738323 learned_addr learned my addr 172.21.15.12:0/171738323 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4979ffb640 1 -- 172.21.15.12:0/171738323 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999420459 0 0) 0x7f497c15f710 con 0x7f497c1601d0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f4979ffb640 1 -- 172.21.15.12:0/171738323 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4968003520 con 0x7f497c1601d0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497affd640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 0x7f497c0cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497b7fe640 1 -- 172.21.15.12:0/171738323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 msgr2=0x7f497c0cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497b7fe640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 0x7f497c0cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497b7fe640 1 -- 172.21.15.12:0/171738323 >> v1:172.21.15.12:6789/0 conn(0x7f497c1601d0 legacy=0x7f497c15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.451+0000 7f497b7fe640 1 -- 172.21.15.12:0/171738323 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c15f9d0 con 0x7f497c0cffb0 2024-03-08T08:29:34.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.452+0000 7f497affd640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 0x7f497c0cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:34.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.452+0000 7f497b7fe640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c0cffb0 0x7f497c15c440 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f496c009d90 tx=0x7f496c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e311efc1d0ff42d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.452+0000 7f4979ffb640 1 -- 172.21.15.12:0/171738323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c039070 con 0x7f497c0cffb0 2024-03-08T08:29:34.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.452+0000 7f4979ffb640 1 -- 172.21.15.12:0/171738323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c02cbc0 con 0x7f497c0cffb0 2024-03-08T08:29:34.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.452+0000 7f4979ffb640 1 -- 172.21.15.12:0/171738323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c034440 con 0x7f497c0cffb0 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 -- 172.21.15.12:0/171738323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c0cffb0 msgr2=0x7f497c15c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c0cffb0 0x7f497c15c440 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f496c009d90 tx=0x7f496c02be10 comp rx=0 tx=0).stop 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 -- 172.21.15.12:0/171738323 shutdown_connections 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c0cffb0 0x7f497c15c440 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 --2- 172.21.15.12:0/171738323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c15b950 0x7f497c0cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 -- 172.21.15.12:0/171738323 >> 172.21.15.12:0/171738323 conn(0x7f497c099a40 msgr2=0x7f497c0b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 -- 172.21.15.12:0/171738323 shutdown_connections 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.453+0000 7f4981895640 1 -- 172.21.15.12:0/171738323 wait complete. 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.454+0000 7f4981895640 1 Processor -- start 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.454+0000 7f4981895640 1 -- start start 2024-03-08T08:29:34.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 0x7f497c0c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c1601d0 0x7f497c154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15cb00 con 0x7f497c15b950 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15c980 con 0x7f497c0cffb0 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f4981895640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0cc2a0 con 0x7f497c1601d0 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44230/0 (socket says 172.21.15.12:44230) 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 -- 172.21.15.12:0/178674487 learned_addr learned my addr 172.21.15.12:0/178674487 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497b7fe640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c1601d0 0x7f497c154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 -- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c1601d0 msgr2=0x7f497c154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c1601d0 0x7f497c154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497affd640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 0x7f497c0c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 -- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 msgr2=0x7f497c0c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 0x7f497c0c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497a7fc640 1 -- 172.21.15.12:0/178674487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0c4830 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.455+0000 7f497affd640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 0x7f497c0c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f497a7fc640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f496400c390 tx=0x7f496400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964017070 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4964005ce0 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c0ae490 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49640072f0 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.456+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0aea50 con 0x7f497c15b950 2024-03-08T08:29:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.457+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c0749c0 con 0x7f497c15b950 2024-03-08T08:29:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.457+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f496401a070 con 0x7f497c15b950 2024-03-08T08:29:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.458+0000 7f4973fff640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 0x7f495c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.458+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49640895d0 con 0x7f497c15b950 2024-03-08T08:29:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.458+0000 7f497affd640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 0x7f495c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.458+0000 7f497affd640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 0x7f495c06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4974006fd0 tx=0x7f4974008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.462+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4964058450 con 0x7f497c15b950 2024-03-08T08:29:34.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.642+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f497c099f20 con 0x7f497c15b950 2024-03-08T08:29:34.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.642+0000 7f4973fff640 1 -- 172.21.15.12:0/178674487 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4964057d50 con 0x7f497c15b950 2024-03-08T08:29:34.643 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 msgr2=0x7f495c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 0x7f495c06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4974006fd0 tx=0x7f4974008040 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 msgr2=0x7f497c0c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f496400c390 tx=0x7f496400c850 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 shutdown_connections 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f495c06b500 0x7f495c06d9c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f497c1601d0 0x7f497c154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c15b950 0x7f497c0c42f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 --2- 172.21.15.12:0/178674487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f497c0cffb0 0x7f497c0c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 >> 172.21.15.12:0/178674487 conn(0x7f497c099a40 msgr2=0x7f497c0b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 shutdown_connections 2024-03-08T08:29:34.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.644+0000 7f4981895640 1 -- 172.21.15.12:0/178674487 wait complete. 2024-03-08T08:29:34.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 Processor -- start 2024-03-08T08:29:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 -- start start 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff10c0d0250 0x7ff10c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 0x7ff10c0d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10c0b6be0 con 0x7ff10c15dec0 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c0d2140 con 0x7ff10c0d0250 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.759+0000 7ff111e95640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c15e2a0 con 0x7ff10c0d25d0 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff10c0d0250 0x7ff10c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff10c0d0250 0x7ff10c0d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44244/0 (socket says 172.21.15.12:44244) 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 -- 172.21.15.12:0/557184926 learned_addr learned my addr 172.21.15.12:0/557184926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 -- 172.21.15.12:0/557184926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 msgr2=0x7ff10c0d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10bfff640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 0x7ff10c0d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 0x7ff10c0d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 -- 172.21.15.12:0/557184926 >> v1:172.21.15.156:6789/0 conn(0x7ff10c15dec0 legacy=0x7ff10c161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10affd640 1 -- 172.21.15.12:0/557184926 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c1620a0 con 0x7ff10c0d0250 2024-03-08T08:29:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.760+0000 7ff10bfff640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 0x7ff10c0d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff10affd640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c0d06b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff10c08ca20 tx=0x7ff0f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177eec15bd7fda1d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff10a7fc640 1 -- 172.21.15.12:0/557184926 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4035020 con 0x7ff10c0d0250 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff10a7fc640 1 -- 172.21.15.12:0/557184926 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f4031830 con 0x7ff10c0d0250 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff10a7fc640 1 -- 172.21.15.12:0/557184926 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4031c00 con 0x7ff10c0d0250 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 -- 172.21.15.12:0/557184926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 msgr2=0x7ff10c0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c0d06b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff10c08ca20 tx=0x7ff0f402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 -- 172.21.15.12:0/557184926 shutdown_connections 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c0d25d0 0x7ff10c0d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 --2- 172.21.15.12:0/557184926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c0d06b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 -- 172.21.15.12:0/557184926 >> 172.21.15.12:0/557184926 conn(0x7ff10c06c240 msgr2=0x7ff10c06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 -- 172.21.15.12:0/557184926 shutdown_connections 2024-03-08T08:29:34.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.761+0000 7ff111e95640 1 -- 172.21.15.12:0/557184926 wait complete. 2024-03-08T08:29:34.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.762+0000 7ff111e95640 1 Processor -- start 2024-03-08T08:29:34.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 -- start start 2024-03-08T08:29:34.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 0x7ff10c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 0x7ff10c15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c162630 con 0x7ff10c0d0250 2024-03-08T08:29:34.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c162350 con 0x7ff10c0d25d0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff111e95640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c0d1dd0 con 0x7ff10c15dec0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff10b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff10b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44250/0 (socket says 172.21.15.12:44250) 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.763+0000 7ff10b7fe640 1 -- 172.21.15.12:0/4292419989 learned_addr learned my addr 172.21.15.12:0/4292419989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10bfff640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 0x7ff10c15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10affd640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 0x7ff10c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 -- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 msgr2=0x7ff10c15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 0x7ff10c15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 -- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 msgr2=0x7ff10c15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 0x7ff10c15bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 -- 172.21.15.12:0/4292419989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c0ca490 con 0x7ff10c0d0250 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10bfff640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 0x7ff10c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10affd640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 0x7ff10c15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff10b7fe640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b3f0 tx=0x7ff0fc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc015020 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc011830 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c0c9bf0 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc011c20 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.764+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c0ca1b0 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.765+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c0749c0 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.766+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0fc018020 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.766+0000 7ff103fff640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 0x7ff0ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.766+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc089ea0 con 0x7ff10c0d0250 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.766+0000 7ff10affd640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 0x7ff0ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:34.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.767+0000 7ff10affd640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 0x7ff0ec06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4036000 tx=0x7ff0f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:34.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.771+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff0fc01f030 con 0x7ff10c0d0250 2024-03-08T08:29:34.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.949+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff10c0b57b0 con 0x7ff10c0d0250 2024-03-08T08:29:34.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.950+0000 7ff103fff640 1 -- 172.21.15.12:0/4292419989 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0fc0172c0 con 0x7ff10c0d0250 2024-03-08T08:29:34.951 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:34.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 msgr2=0x7ff0ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 0x7ff0ec06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4036000 tx=0x7ff0f4008030 comp rx=0 tx=0).stop 2024-03-08T08:29:34.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 msgr2=0x7ff10c14be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b3f0 tx=0x7ff0fc00b8b0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 shutdown_connections 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff0ec06b4d0 0x7ff0ec06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff10c15dec0 0x7ff10c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff10c0d25d0 0x7ff10c15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 --2- 172.21.15.12:0/4292419989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff10c0d0250 0x7ff10c14be40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 >> 172.21.15.12:0/4292419989 conn(0x7ff10c06c240 msgr2=0x7ff10c0b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 shutdown_connections 2024-03-08T08:29:34.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:34.952+0000 7ff111e95640 1 -- 172.21.15.12:0/4292419989 wait complete. 2024-03-08T08:29:34.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:35.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.064+0000 7f73343c7640 1 Processor -- start 2024-03-08T08:29:35.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.064+0000 7f73343c7640 1 -- start start 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.065+0000 7f73343c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f732c099c00 0x7f732c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.065+0000 7f73343c7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.065+0000 7f73343c7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c09a520 con 0x7f732c0b24c0 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.065+0000 7f73343c7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732c09ad80 con 0x7f732c099c00 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.065+0000 7f73343c7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0cf570 con 0x7f732c09a880 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7331962640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f732c099c00 0x7f732c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f732c099c00 0x7f732c0c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53172/0 (socket says 172.21.15.12:53172) 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 -- 172.21.15.12:0/3806678165 learned_addr learned my addr 172.21.15.12:0/3806678165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7331962640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:47946/0 (socket says 172.21.15.12:47946) 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332964640 1 --1- 172.21.15.12:0/3806678165 >> v1:172.21.15.156:6789/0 conn(0x7f732c0b24c0 0x7f732c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:41376/0 (socket says 172.21.15.12:41376) 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 -- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 msgr2=0x7f732c099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 -- 172.21.15.12:0/3806678165 >> v1:172.21.15.156:6789/0 conn(0x7f732c0b24c0 legacy=0x7f732c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7332163640 1 -- 172.21.15.12:0/3806678165 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c0cf850 con 0x7f732c099c00 2024-03-08T08:29:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.066+0000 7f7331962640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f7332163640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c099c00 0x7f732c0c54f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7324009e50 tx=0x7f732402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a36293b62c178964 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f7331161640 1 -- 172.21.15.12:0/3806678165 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732402cb00 con 0x7f732c099c00 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f7331161640 1 -- 172.21.15.12:0/3806678165 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732402cca0 con 0x7f732c099c00 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f7331161640 1 -- 172.21.15.12:0/3806678165 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324034610 con 0x7f732c099c00 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f73343c7640 1 -- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c099c00 msgr2=0x7f732c0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.067+0000 7f73343c7640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c099c00 0x7f732c0c54f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7324009e50 tx=0x7f732402bf60 comp rx=0 tx=0).stop 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 -- 172.21.15.12:0/3806678165 shutdown_connections 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c09a880 0x7f732c099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 --2- 172.21.15.12:0/3806678165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c099c00 0x7f732c0c54f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 -- 172.21.15.12:0/3806678165 >> 172.21.15.12:0/3806678165 conn(0x7f732c0b3490 msgr2=0x7f732c0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 -- 172.21.15.12:0/3806678165 shutdown_connections 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.068+0000 7f73343c7640 1 -- 172.21.15.12:0/3806678165 wait complete. 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 Processor -- start 2024-03-08T08:29:35.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 -- start start 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c09a880 0x7f732c151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 0x7f732c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0b2930 con 0x7f732c099c00 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0ce0c0 con 0x7f732c09a880 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.069+0000 7f73343c7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1f6380 con 0x7f732c0b24c0 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44262/0 (socket says 172.21.15.12:44262) 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 -- 172.21.15.12:0/2722875491 learned_addr learned my addr 172.21.15.12:0/2722875491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7331962640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c09a880 0x7f732c151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332964640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 0x7f732c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 -- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 msgr2=0x7f732c070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 0x7f732c070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 -- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c09a880 msgr2=0x7f732c151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c09a880 0x7f732c151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 -- 172.21.15.12:0/2722875491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c0b5960 con 0x7f732c099c00 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332964640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 0x7f732c070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f7332163640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7324036040 tx=0x7f7324004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324043020 con 0x7f732c099c00 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732402ce50 con 0x7f732c099c00 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c0b5c40 con 0x7f732c099c00 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324009510 con 0x7f732c099c00 2024-03-08T08:29:35.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.070+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c0b6200 con 0x7f732c099c00 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.071+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c0749c0 con 0x7f732c099c00 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.072+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7324046020 con 0x7f732c099c00 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.072+0000 7f732affd640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 0x7f731806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.072+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73240ac680 con 0x7f732c099c00 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.072+0000 7f7331962640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 0x7f731806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.073+0000 7f7331962640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 0x7f731806d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7320009560 tx=0x7f7320009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.076+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f732404d030 con 0x7f732c099c00 2024-03-08T08:29:35.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.257+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f732c0b42f0 con 0x7f732c099c00 2024-03-08T08:29:35.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.258+0000 7f732affd640 1 -- 172.21.15.12:0/2722875491 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f732407b590 con 0x7f732c099c00 2024-03-08T08:29:35.259 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:35.259 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 msgr2=0x7f731806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 0x7f731806d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7320009560 tx=0x7f7320009190 comp rx=0 tx=0).stop 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 msgr2=0x7f732c1f5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7324036040 tx=0x7f7324004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 shutdown_connections 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731806b500 0x7f731806d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f732c0b24c0 0x7f732c070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f732c09a880 0x7f732c151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 --2- 172.21.15.12:0/2722875491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f732c099c00 0x7f732c1f5e40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 >> 172.21.15.12:0/2722875491 conn(0x7f732c0b3490 msgr2=0x7f732c15edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 shutdown_connections 2024-03-08T08:29:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.260+0000 7f73343c7640 1 -- 172.21.15.12:0/2722875491 wait complete. 2024-03-08T08:29:35.272 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:35.272 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:35.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.376+0000 7f17520ae640 1 Processor -- start 2024-03-08T08:29:35.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.376+0000 7f17520ae640 1 -- start start 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f17520ae640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f174c09a9c0 0x7f174c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f17520ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c0c5a30 0x7f174c15f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f17520ae640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c15c090 con 0x7f174c15bcb0 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f17520ae640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c0cd7a0 con 0x7f174c09a9c0 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f17520ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c099c00 con 0x7f174c0c5a30 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f174affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f174c09a9c0 0x7f174c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f174affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f174c09a9c0 0x7f174c0c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44272/0 (socket says 172.21.15.12:44272) 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.377+0000 7f174affd640 1 -- 172.21.15.12:0/2865428283 learned_addr learned my addr 172.21.15.12:0/2865428283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174affd640 1 -- 172.21.15.12:0/2865428283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c0c5a30 msgr2=0x7f174c15f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174affd640 1 --2- 172.21.15.12:0/2865428283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c0c5a30 0x7f174c15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174affd640 1 -- 172.21.15.12:0/2865428283 >> v1:172.21.15.156:6789/0 conn(0x7f174c15bcb0 legacy=0x7f174c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174affd640 1 -- 172.21.15.12:0/2865428283 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c099ee0 con 0x7f174c09a9c0 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174affd640 1 --2- 172.21.15.12:0/2865428283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c09a9c0 0x7f174c0c54f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1740009870 tx=0x7f174002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adc53d44297bf32 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174a7fc640 1 -- 172.21.15.12:0/2865428283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17400317c0 con 0x7f174c09a9c0 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174a7fc640 1 -- 172.21.15.12:0/2865428283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1740037070 con 0x7f174c09a9c0 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.378+0000 7f174a7fc640 1 -- 172.21.15.12:0/2865428283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740031b80 con 0x7f174c09a9c0 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 -- 172.21.15.12:0/2865428283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c09a9c0 msgr2=0x7f174c0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 --2- 172.21.15.12:0/2865428283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c09a9c0 0x7f174c0c54f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1740009870 tx=0x7f174002b860 comp rx=0 tx=0).stop 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 -- 172.21.15.12:0/2865428283 shutdown_connections 2024-03-08T08:29:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 --2- 172.21.15.12:0/2865428283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c0c5a30 0x7f174c15f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 --2- 172.21.15.12:0/2865428283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c09a9c0 0x7f174c0c54f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 -- 172.21.15.12:0/2865428283 >> 172.21.15.12:0/2865428283 conn(0x7f174c0b3490 msgr2=0x7f174c0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 -- 172.21.15.12:0/2865428283 shutdown_connections 2024-03-08T08:29:35.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.379+0000 7f17520ae640 1 -- 172.21.15.12:0/2865428283 wait complete. 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.380+0000 7f17520ae640 1 Processor -- start 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 -- start start 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c09a9c0 0x7f174c152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f174c0c5a30 0x7f174c0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c09ae50 con 0x7f174c15bcb0 2024-03-08T08:29:35.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c09a4a0 con 0x7f174c0c5a30 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.381+0000 7f17520ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c06caa0 con 0x7f174c09a9c0 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44278/0 (socket says 172.21.15.12:44278) 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 -- 172.21.15.12:0/1961276845 learned_addr learned my addr 172.21.15.12:0/1961276845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174affd640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f174c0c5a30 0x7f174c0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174b7fe640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c09a9c0 0x7f174c152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 -- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c09a9c0 msgr2=0x7f174c152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c09a9c0 0x7f174c152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 -- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f174c0c5a30 msgr2=0x7f174c0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f174c0c5a30 0x7f174c0c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 -- 172.21.15.12:0/1961276845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c06cd80 con 0x7f174c15bcb0 2024-03-08T08:29:35.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f174bfff640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f174400b970 tx=0x7f174400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744019070 con 0x7f174c15bcb0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174400cb60 con 0x7f174c15bcb0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c06d060 con 0x7f174c15bcb0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744014440 con 0x7f174c15bcb0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.382+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c06d620 con 0x7f174c15bcb0 2024-03-08T08:29:35.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.383+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174c0749c0 con 0x7f174c15bcb0 2024-03-08T08:29:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.384+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f174401c070 con 0x7f174c15bcb0 2024-03-08T08:29:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.384+0000 7f1733fff640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 0x7f173806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.384+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1744089e70 con 0x7f174c15bcb0 2024-03-08T08:29:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.384+0000 7f174b7fe640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 0x7f173806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.385+0000 7f174b7fe640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 0x7f173806d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f174c15f310 tx=0x7f173c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.392+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f17440586a0 con 0x7f174c15bcb0 2024-03-08T08:29:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.572+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f174c0cf6e0 con 0x7f174c15bcb0 2024-03-08T08:29:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.572+0000 7f1733fff640 1 -- 172.21.15.12:0/1961276845 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f174405c740 con 0x7f174c15bcb0 2024-03-08T08:29:35.574 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 msgr2=0x7f173806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 0x7f173806d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f174c15f310 tx=0x7f173c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 msgr2=0x7f174c0c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f174400b970 tx=0x7f174400be30 comp rx=0 tx=0).stop 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 shutdown_connections 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.574+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f173806b4d0 0x7f173806d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f174c15bcb0 0x7f174c0c9970 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f174c0c5a30 0x7f174c0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 --2- 172.21.15.12:0/1961276845 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f174c09a9c0 0x7f174c152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 >> 172.21.15.12:0/1961276845 conn(0x7f174c0b3490 msgr2=0x7f174c0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 shutdown_connections 2024-03-08T08:29:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.575+0000 7f17520ae640 1 -- 172.21.15.12:0/1961276845 wait complete. 2024-03-08T08:29:35.587 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:35.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.689+0000 7feddbc9a640 1 Processor -- start 2024-03-08T08:29:35.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 -- start start 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd415bd00 0x7fedd4099b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 0x7fedd40c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd415c0e0 con 0x7fedd409a060 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd409a4c0 con 0x7fedd415bd00 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7feddbc9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd40cd210 con 0x7fedd40ccda0 2024-03-08T08:29:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7fedd9a36640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd415bd00 0x7fedd4099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7fedd9a36640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd415bd00 0x7fedd4099b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44280/0 (socket says 172.21.15.12:44280) 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.690+0000 7fedd9a36640 1 -- 172.21.15.12:0/1163534361 learned_addr learned my addr 172.21.15.12:0/1163534361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedda237640 1 --2- 172.21.15.12:0/1163534361 >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 0x7fedd40c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedd9a36640 1 -- 172.21.15.12:0/1163534361 >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 msgr2=0x7fedd40c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedd9a36640 1 --2- 172.21.15.12:0/1163534361 >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 0x7fedd40c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedd9a36640 1 -- 172.21.15.12:0/1163534361 >> v1:172.21.15.156:6789/0 conn(0x7fedd409a060 legacy=0x7fedd40cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedd9a36640 1 -- 172.21.15.12:0/1163534361 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40cd450 con 0x7fedd415bd00 2024-03-08T08:29:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedda237640 1 --2- 172.21.15.12:0/1163534361 >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 0x7fedd40c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:35.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.691+0000 7fedd9a36640 1 --2- 172.21.15.12:0/1163534361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd415bd00 0x7fedd4099b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fedd404ff70 tx=0x7fedcc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c01b4859817589 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7fedd8a34640 1 -- 172.21.15.12:0/1163534361 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc034070 con 0x7fedd415bd00 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7fedd8a34640 1 -- 172.21.15.12:0/1163534361 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc030860 con 0x7fedd415bd00 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7fedd8a34640 1 -- 172.21.15.12:0/1163534361 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc030c30 con 0x7fedd415bd00 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 -- 172.21.15.12:0/1163534361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd415bd00 msgr2=0x7fedd4099b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 --2- 172.21.15.12:0/1163534361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd415bd00 0x7fedd4099b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fedd404ff70 tx=0x7fedcc02b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 -- 172.21.15.12:0/1163534361 shutdown_connections 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 --2- 172.21.15.12:0/1163534361 >> v2:172.21.15.156:3300/0 conn(0x7fedd40ccda0 0x7fedd40c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 --2- 172.21.15.12:0/1163534361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd415bd00 0x7fedd4099b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 -- 172.21.15.12:0/1163534361 >> 172.21.15.12:0/1163534361 conn(0x7fedd40b3470 msgr2=0x7fedd40b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 -- 172.21.15.12:0/1163534361 shutdown_connections 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.692+0000 7feddbc9a640 1 -- 172.21.15.12:0/1163534361 wait complete. 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.693+0000 7feddbc9a640 1 Processor -- start 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.693+0000 7feddbc9a640 1 -- start start 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd40ccda0 0x7fedd4157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd415bd00 0x7fedd40cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40c5950 con 0x7fedd409a060 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40c5670 con 0x7fedd40ccda0 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7feddbc9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40c63b0 con 0x7fedd415bd00 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44290/0 (socket says 172.21.15.12:44290) 2024-03-08T08:29:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 -- 172.21.15.12:0/2565738676 learned_addr learned my addr 172.21.15.12:0/2565738676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 -- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd415bd00 msgr2=0x7fedd40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd415bd00 0x7fedd40cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 -- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd40ccda0 msgr2=0x7fedd4157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd40ccda0 0x7fedd4157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.694+0000 7fedd9a36640 1 -- 172.21.15.12:0/2565738676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40b4740 con 0x7fedd409a060 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7fedd9a36640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fedcc000c00 tx=0x7fedcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc034040 con 0x7fedd409a060 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc039d90 con 0x7fedd409a060 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd40b4e50 con 0x7fedd409a060 2024-03-08T08:29:35.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc043610 con 0x7fedd409a060 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.695+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd40b5410 con 0x7fedd409a060 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.696+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd40749c0 con 0x7fedd409a060 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.696+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedcc037070 con 0x7fedd409a060 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.697+0000 7fedd27fc640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.697+0000 7fedd9235640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.697+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedcc0ac3d0 con 0x7fedd409a060 2024-03-08T08:29:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.697+0000 7fedd9235640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fedc8000f60 tx=0x7fedc80079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:35.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.701+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedcc03e070 con 0x7fedd409a060 2024-03-08T08:29:35.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.882+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fedd415dab0 con 0x7fedd409a060 2024-03-08T08:29:35.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.882+0000 7fedd27fc640 1 -- 172.21.15.12:0/2565738676 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fedcc07b370 con 0x7fedd409a060 2024-03-08T08:29:35.884 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 msgr2=0x7fedc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fedc8000f60 tx=0x7fedc80079b0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 msgr2=0x7fedd40b73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fedcc000c00 tx=0x7fedcc0046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 shutdown_connections 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd415bd00 0x7fedd40cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd40ccda0 0x7fedd4157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 --2- 172.21.15.12:0/2565738676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd409a060 0x7fedd40b73e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:35.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 >> 172.21.15.12:0/2565738676 conn(0x7fedd40b3470 msgr2=0x7fedd40c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:35.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 shutdown_connections 2024-03-08T08:29:35.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.884+0000 7feddbc9a640 1 -- 172.21.15.12:0/2565738676 wait complete. 2024-03-08T08:29:35.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:36.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:35.999+0000 7f9f08572640 1 Processor -- start 2024-03-08T08:29:36.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.000+0000 7f9f08572640 1 -- start start 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.000+0000 7f9f08572640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f0015e0d0 0x7f9f000cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.000+0000 7f9f08572640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f000d0290 con 0x7f9f00099b90 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.000+0000 7f9f08572640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f000d0570 con 0x7f9f0006bef0 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.000+0000 7f9f08572640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f000d0850 con 0x7f9f0015e0d0 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f0015e0d0 0x7f9f000cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9f0015e0d0 0x7f9f000cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44300/0 (socket says 172.21.15.12:44300) 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f0630e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9f00099b90 0x7f9f00099f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46372/0 (socket says 172.21.15.12:46372) 2024-03-08T08:29:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 -- 172.21.15.12:0/1066216986 learned_addr learned my addr 172.21.15.12:0/1066216986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 -- 172.21.15.12:0/1066216986 >> v1:172.21.15.156:6789/0 conn(0x7f9f0006bef0 legacy=0x7f9f000cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 -- 172.21.15.12:0/1066216986 >> v1:172.21.15.12:6789/0 conn(0x7f9f00099b90 legacy=0x7f9f00099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f05b0d640 1 -- 172.21.15.12:0/1066216986 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f000d0b30 con 0x7f9f0015e0d0 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.001+0000 7f9f0530c640 1 -- 172.21.15.12:0/1066216986 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446536678 0 0) 0x7f9f000d0290 con 0x7f9f00099b90 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f05b0d640 1 --2- 172.21.15.12:0/1066216986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0015e0d0 0x7f9f000cfc40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009870 tx=0x7f9ef402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f8d0c41ac1ede6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f0530c640 1 -- 172.21.15.12:0/1066216986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4035020 con 0x7f9f0015e0d0 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f0530c640 1 -- 172.21.15.12:0/1066216986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef4031870 con 0x7f9f0015e0d0 2024-03-08T08:29:36.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f0530c640 1 -- 172.21.15.12:0/1066216986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4031c40 con 0x7f9f0015e0d0 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 -- 172.21.15.12:0/1066216986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0015e0d0 msgr2=0x7f9f000cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 --2- 172.21.15.12:0/1066216986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0015e0d0 0x7f9f000cfc40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009870 tx=0x7f9ef402b910 comp rx=0 tx=0).stop 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 -- 172.21.15.12:0/1066216986 shutdown_connections 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 --2- 172.21.15.12:0/1066216986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0015e0d0 0x7f9f000cfc40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 -- 172.21.15.12:0/1066216986 >> 172.21.15.12:0/1066216986 conn(0x7f9f000b3460 msgr2=0x7f9f000b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 -- 172.21.15.12:0/1066216986 shutdown_connections 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.002+0000 7f9f08572640 1 -- 172.21.15.12:0/1066216986 wait complete. 2024-03-08T08:29:36.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 Processor -- start 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 -- start start 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f00099b90 0x7f9f000af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f000afea0 0x7f9f000ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0009b310 con 0x7f9f0006bef0 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0009b190 con 0x7f9f000afea0 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f08572640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f000d1330 con 0x7f9f00099b90 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f0630e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f0630e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44304/0 (socket says 172.21.15.12:44304) 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.004+0000 7f9f0630e640 1 -- 172.21.15.12:0/3608403919 learned_addr learned my addr 172.21.15.12:0/3608403919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f05b0d640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f00099b90 0x7f9f000af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f06b0f640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f000afea0 0x7f9f000ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 -- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f00099b90 msgr2=0x7f9f000af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f00099b90 0x7f9f000af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 -- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f000afea0 msgr2=0x7f9f000ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f000afea0 0x7f9f000ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 -- 172.21.15.12:0/3608403919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f000cb630 con 0x7f9f0006bef0 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f0630e640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c4d0 tx=0x7f9efc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc018070 con 0x7f9f0006bef0 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc007500 con 0x7f9f0006bef0 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0015bb20 con 0x7f9f0006bef0 2024-03-08T08:29:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc004a20 con 0x7f9f0006bef0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.005+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0015c0e0 con 0x7f9f0006bef0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.006+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f000b02b0 con 0x7f9f0006bef0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.006+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9efc01b070 con 0x7f9f0006bef0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.007+0000 7f9ef2ffd640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 0x7f9ee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.007+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9efc089570 con 0x7f9f0006bef0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.007+0000 7f9f05b0d640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 0x7f9ee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.007+0000 7f9f05b0d640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 0x7f9ee406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4036000 tx=0x7f9ef403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.011+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f000b02b0 con 0x7f9f0006bef0 2024-03-08T08:29:36.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.191+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f000749c0 con 0x7f9f0006bef0 2024-03-08T08:29:36.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.192+0000 7f9ef2ffd640 1 -- 172.21.15.12:0/3608403919 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9efc087e20 con 0x7f9f0006bef0 2024-03-08T08:29:36.193 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.193+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 msgr2=0x7f9ee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.193+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 0x7f9ee406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4036000 tx=0x7f9ef403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 msgr2=0x7f9f001f0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c4d0 tx=0x7f9efc00c990 comp rx=0 tx=0).stop 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 shutdown_connections 2024-03-08T08:29:36.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ee406b4d0 0x7f9ee406d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f000afea0 0x7f9f000ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f00099b90 0x7f9f000af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 --2- 172.21.15.12:0/3608403919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f0006bef0 0x7f9f001f0df0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 >> 172.21.15.12:0/3608403919 conn(0x7f9f000b3460 msgr2=0x7f9f00099f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 shutdown_connections 2024-03-08T08:29:36.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.194+0000 7f9f08572640 1 -- 172.21.15.12:0/3608403919 wait complete. 2024-03-08T08:29:36.225 INFO:tasks.cephfs_test_runner:Starting test: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) That the per-client labelled perf counters depict the clients 2024-03-08T08:29:36.225 INFO:tasks.cephfs_test_runner:test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) 2024-03-08T08:29:36.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.326+0000 7fa019da2640 1 Processor -- start 2024-03-08T08:29:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.326+0000 7fa019da2640 1 -- start start 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa019da2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa019da2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0140b24b0 0x7fa01406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa019da2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0140d14f0 con 0x7fa0140d4500 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa019da2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0140d17d0 con 0x7fa0140b24b0 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa019da2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140d1ab0 con 0x7fa014160f70 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa0195a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0140b24b0 0x7fa01406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa0195a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0140b24b0 0x7fa01406c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44308/0 (socket says 172.21.15.12:44308) 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa0195a1640 1 -- 172.21.15.12:0/602567136 learned_addr learned my addr 172.21.15.12:0/602567136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa0195a1640 1 -- 172.21.15.12:0/602567136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 msgr2=0x7fa01406bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.327+0000 7fa0195a1640 1 --2- 172.21.15.12:0/602567136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.328+0000 7fa0195a1640 1 -- 172.21.15.12:0/602567136 >> v1:172.21.15.156:6789/0 conn(0x7fa0140d4500 legacy=0x7fa014161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.328+0000 7fa0195a1640 1 -- 172.21.15.12:0/602567136 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0140d1d90 con 0x7fa0140b24b0 2024-03-08T08:29:36.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.328+0000 7fa0195a1640 1 --2- 172.21.15.12:0/602567136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa01406c4f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ab3e47f2623f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.330+0000 7fa0137fe640 1 -- 172.21.15.12:0/602567136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc035020 con 0x7fa0140b24b0 2024-03-08T08:29:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.330+0000 7fa0137fe640 1 -- 172.21.15.12:0/602567136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc031640 con 0x7fa0140b24b0 2024-03-08T08:29:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.330+0000 7fa0137fe640 1 -- 172.21.15.12:0/602567136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc02d040 con 0x7fa0140b24b0 2024-03-08T08:29:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 -- 172.21.15.12:0/602567136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 msgr2=0x7fa01406c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 --2- 172.21.15.12:0/602567136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa01406c4f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc02b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 -- 172.21.15.12:0/602567136 shutdown_connections 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 --2- 172.21.15.12:0/602567136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa01406c4f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 --2- 172.21.15.12:0/602567136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 -- 172.21.15.12:0/602567136 >> 172.21.15.12:0/602567136 conn(0x7fa0140d09d0 msgr2=0x7fa0140d3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 -- 172.21.15.12:0/602567136 shutdown_connections 2024-03-08T08:29:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.331+0000 7fa019da2640 1 -- 172.21.15.12:0/602567136 wait complete. 2024-03-08T08:29:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.332+0000 7fa019da2640 1 Processor -- start 2024-03-08T08:29:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.332+0000 7fa019da2640 1 -- start start 2024-03-08T08:29:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0140d4500 0x7fa01415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01415f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140b3050 con 0x7fa0140b24b0 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140b6a50 con 0x7fa0140d4500 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa019da2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140b31d0 con 0x7fa014160f70 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44324/0 (socket says 172.21.15.12:44324) 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 -- 172.21.15.12:0/827565443 learned_addr learned my addr 172.21.15.12:0/827565443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa013fff640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0140d4500 0x7fa01415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa0195a1640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01415f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 -- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 msgr2=0x7fa01415f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01415f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 -- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0140d4500 msgr2=0x7fa01415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0140d4500 0x7fa01415c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa018da0640 1 -- 172.21.15.12:0/827565443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0141f84c0 con 0x7fa0140b24b0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.333+0000 7fa0195a1640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01415f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa018da0640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00b410 tx=0x7fa00c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c015020 con 0x7fa0140b24b0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c011850 con 0x7fa0140b24b0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa014159940 con 0x7fa0140b24b0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c011c40 con 0x7fa0140b24b0 2024-03-08T08:29:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.334+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa014159f00 con 0x7fa0140b24b0 2024-03-08T08:29:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.335+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0140749c0 con 0x7fa0140b24b0 2024-03-08T08:29:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.335+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa00c018020 con 0x7fa0140b24b0 2024-03-08T08:29:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.336+0000 7fa0117fa640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.336+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00c08a0c0 con 0x7fa0140b24b0 2024-03-08T08:29:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.336+0000 7fa013fff640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.336+0000 7fa013fff640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0140b2e00 tx=0x7fa004009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.340+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa00c01f020 con 0x7fa0140b24b0 2024-03-08T08:29:36.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.520+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0140ca4d0 con 0x7fa0140b24b0 2024-03-08T08:29:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.521+0000 7fa0117fa640 1 -- 172.21.15.12:0/827565443 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fa0141f84c0 con 0x7fa0140b24b0 2024-03-08T08:29:36.522 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:36.522 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.522+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 msgr2=0x7f9ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.522+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0140b2e00 tx=0x7fa004009040 comp rx=0 tx=0).stop 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.522+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 msgr2=0x7fa0141f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.522+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00b410 tx=0x7fa00c00b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 shutdown_connections 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa014160f70 0x7fa01415f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0140d4500 0x7fa01415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 --2- 172.21.15.12:0/827565443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0140b24b0 0x7fa0141f7f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 >> 172.21.15.12:0/827565443 conn(0x7fa0140d09d0 msgr2=0x7fa0140b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 shutdown_connections 2024-03-08T08:29:36.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.523+0000 7fa019da2640 1 -- 172.21.15.12:0/827565443 wait complete. 2024-03-08T08:29:36.535 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:36.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.638+0000 7f9a8f979640 1 Processor -- start 2024-03-08T08:29:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.638+0000 7f9a8f979640 1 -- start start 2024-03-08T08:29:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.638+0000 7f9a8f979640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9a880d0250 0x7f9a880d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.638+0000 7f9a8f979640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880b6be0 con 0x7f9a880d25d0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8f979640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8815e2a0 con 0x7f9a8815dec0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8f979640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880d2140 con 0x7f9a880d0250 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8df16640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9a880d25d0 0x7f9a880d2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46382/0 (socket says 172.21.15.12:46382) 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8df16640 1 -- 172.21.15.12:0/4158393455 learned_addr learned my addr 172.21.15.12:0/4158393455 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42560410 0 0) 0x7f9a78001490 con 0x7f9a880d25d0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a70003520 con 0x7f9a880d25d0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/4158393455 >> v2:172.21.15.156:3300/0 conn(0x7f9a880d0250 0x7f9a880d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1249456773 0 0) 0x7f9a70003520 con 0x7f9a880d25d0 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 >> v2:172.21.15.156:3300/0 conn(0x7f9a880d0250 msgr2=0x7f9a880d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 --2- 172.21.15.12:0/4158393455 >> v2:172.21.15.156:3300/0 conn(0x7f9a880d0250 0x7f9a880d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 >> v1:172.21.15.156:6789/0 conn(0x7f9a8815dec0 legacy=0x7f9a88161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a881620a0 con 0x7f9a880d25d0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f9a78001bc0 con 0x7f9a880d25d0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 >> v1:172.21.15.12:6789/0 conn(0x7f9a880d25d0 legacy=0x7f9a880d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a700039b0 0x7f9a70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a70004800 0x7f9a70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880d2140 con 0x7f9a700051c0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7000c060 con 0x7f9a700039b0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7000c260 con 0x7f9a70004800 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.639+0000 7f9a8df16640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8d715640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a700039b0 0x7f9a70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a70004800 0x7f9a70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 -- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a70004800 msgr2=0x7f9a70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a70004800 0x7f9a70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 -- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a700039b0 msgr2=0x7f9a70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a700039b0 0x7f9a70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 -- 172.21.15.12:0/4158393455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70003520 con 0x7f9a700051c0 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a70004800 0x7f9a70004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 -- 172.21.15.12:0/4158393455 reap_dead start 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.640+0000 7f9a8df16640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a7000b9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800aa50 tx=0x7f9a70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f35f33a410fb07d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78038070 con 0x7f9a700051c0 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78033650 con 0x7f9a700051c0 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a87fff640 1 -- 172.21.15.12:0/4158393455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800a670 con 0x7f9a700051c0 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 -- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 msgr2=0x7f9a7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a7000b9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800aa50 tx=0x7f9a70005830 comp rx=0 tx=0).stop 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 -- 172.21.15.12:0/4158393455 shutdown_connections 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 --2- 172.21.15.12:0/4158393455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a7000b9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 -- 172.21.15.12:0/4158393455 >> 172.21.15.12:0/4158393455 conn(0x7f9a8806c240 msgr2=0x7f9a8806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 -- 172.21.15.12:0/4158393455 shutdown_connections 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.641+0000 7f9a8f979640 1 -- 172.21.15.12:0/4158393455 wait complete. 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.642+0000 7f9a8f979640 1 Processor -- start 2024-03-08T08:29:36.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.642+0000 7f9a8f979640 1 -- start start 2024-03-08T08:29:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a880b5e90 0x7f9a88160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a88160ba0 0x7f9a880cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880d1dd0 con 0x7f9a700051c0 2024-03-08T08:29:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88162630 con 0x7f9a880b5e90 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8f979640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880b94f0 con 0x7f9a88160ba0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44348/0 (socket says 172.21.15.12:44348) 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 -- 172.21.15.12:0/2928791810 learned_addr learned my addr 172.21.15.12:0/2928791810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a880b5e90 0x7f9a88160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8df16640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a88160ba0 0x7f9a880cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 -- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a88160ba0 msgr2=0x7f9a880cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a88160ba0 0x7f9a880cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 -- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a880b5e90 msgr2=0x7f9a88160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a880b5e90 0x7f9a88160660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.643+0000 7f9a8d715640 1 -- 172.21.15.12:0/2928791810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880b3fc0 con 0x7f9a700051c0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a8d715640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000f5f0 tx=0x7f9a8000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800053a0 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80018070 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8814f2a0 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003830 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.644+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880b4540 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.645+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a880749c0 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.645+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a800039d0 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.646+0000 7f9a85ffb640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 0x7f9a7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.646+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 0x7f9a7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.646+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a8008a070 con 0x7f9a700051c0 2024-03-08T08:29:36.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.646+0000 7f9a8cf14640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 0x7f9a7406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9a880b2700 tx=0x7f9a7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.650+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9a80058e70 con 0x7f9a700051c0 2024-03-08T08:29:36.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.829+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a88160f80 con 0x7f9a700051c0 2024-03-08T08:29:36.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.829+0000 7f9a85ffb640 1 -- 172.21.15.12:0/2928791810 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a80058770 con 0x7f9a700051c0 2024-03-08T08:29:36.830 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 msgr2=0x7f9a7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 0x7f9a7406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9a880b2700 tx=0x7f9a7c008040 comp rx=0 tx=0).stop 2024-03-08T08:29:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 msgr2=0x7f9a8814ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000f5f0 tx=0x7f9a8000fab0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 shutdown_connections 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9a7406b4d0 0x7f9a7406d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a88160ba0 0x7f9a880cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a880b5e90 0x7f9a88160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 --2- 172.21.15.12:0/2928791810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a700051c0 0x7f9a8814ed60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 >> 172.21.15.12:0/2928791810 conn(0x7f9a8806c240 msgr2=0x7f9a880c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 shutdown_connections 2024-03-08T08:29:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.831+0000 7f9a8f979640 1 -- 172.21.15.12:0/2928791810 wait complete. 2024-03-08T08:29:36.843 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:36.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.946+0000 7fbcbfcc0640 1 Processor -- start 2024-03-08T08:29:36.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.946+0000 7fbcbfcc0640 1 -- start start 2024-03-08T08:29:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbfcc0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbcb80d0250 0x7fbcb80d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbfcc0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb80b6be0 con 0x7fbcb815dec0 2024-03-08T08:29:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbfcc0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb815e2a0 con 0x7fbcb80d25d0 2024-03-08T08:29:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbfcc0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb80d2140 con 0x7fbcb80d0250 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbda5c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbcb815dec0 0x7fbcb8161990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46398/0 (socket says 172.21.15.12:46398) 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbda5c640 1 -- 172.21.15.12:0/1127818685 learned_addr learned my addr 172.21.15.12:0/1127818685 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177487709 0 0) 0x7fbcb4001110 con 0x7fbcb815dec0 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca4003520 con 0x7fbcb815dec0 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/1127818685 >> v2:172.21.15.156:3300/0 conn(0x7fbcb80d0250 0x7fbcb80d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656816396 0 0) 0x7fbca4003520 con 0x7fbcb815dec0 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 >> v2:172.21.15.156:3300/0 conn(0x7fbcb80d0250 msgr2=0x7fbcb80d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 --2- 172.21.15.12:0/1127818685 >> v2:172.21.15.156:3300/0 conn(0x7fbcb80d0250 0x7fbcb80d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 >> v1:172.21.15.156:6789/0 conn(0x7fbcb80d25d0 legacy=0x7fbcb80d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb81620a0 con 0x7fbcb815dec0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fbcb4001840 con 0x7fbcb815dec0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 >> v1:172.21.15.12:6789/0 conn(0x7fbcb815dec0 legacy=0x7fbcb8161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbca4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4004800 0x7fbca4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40051c0 0x7fbca400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80d2140 con 0x7fbca40039b0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca400c060 con 0x7fbca4004800 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca400c260 con 0x7fbca40051c0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.947+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbca4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4004800 0x7fbca4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbe25d640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40051c0 0x7fbca400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 -- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40051c0 msgr2=0x7fbca400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbca40051c0 0x7fbca400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 -- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4004800 msgr2=0x7fbca4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbca4004800 0x7fbca4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbe25d640 1 -- 172.21.15.12:0/1127818685 reap_dead start 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 -- 172.21.15.12:0/1127818685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4003520 con 0x7fbca40039b0 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.948+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbca4003dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c4c0 tx=0x7fbca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8316ddbe9816e693 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb400c0e0 con 0x7fbca40039b0 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4030070 con 0x7fbca40039b0 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbca5a640 1 -- 172.21.15.12:0/1127818685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40058d0 con 0x7fbca40039b0 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 msgr2=0x7fbca4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbca4003dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c4c0 tx=0x7fbca4005830 comp rx=0 tx=0).stop 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/1127818685 shutdown_connections 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/1127818685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbca4003dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/1127818685 >> 172.21.15.12:0/1127818685 conn(0x7fbcb806c240 msgr2=0x7fbcb806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/1127818685 shutdown_connections 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.949+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/1127818685 wait complete. 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.950+0000 7fbcbfcc0640 1 Processor -- start 2024-03-08T08:29:36.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.950+0000 7fbcbfcc0640 1 -- start start 2024-03-08T08:29:36.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 0x7fbcb8148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 0x7fbcb80ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80d1dd0 con 0x7fbca40039b0 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb8162630 con 0x7fbcb80cc7b0 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbfcc0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b94f0 con 0x7fbcb8148300 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44370/0 (socket says 172.21.15.12:44370) 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 -- 172.21.15.12:0/3244791489 learned_addr learned my addr 172.21.15.12:0/3244791489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbe25d640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 0x7fbcb80ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 0x7fbcb8148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 -- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 msgr2=0x7fbcb8148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 0x7fbcb8148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 -- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 msgr2=0x7fbcb80ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 0x7fbcb80ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbda5c640 1 -- 172.21.15.12:0/3244791489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb80cd4e0 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbe25d640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 0x7fbcb80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.951+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 0x7fbcb8148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcbda5c640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c4c0 tx=0x7fbcb402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40049e0 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4004b80 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb80cd7c0 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4007070 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.952+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb8149360 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.953+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb80cfd30 con 0x7fbca40039b0 2024-03-08T08:29:36.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.953+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcb4007210 con 0x7fbca40039b0 2024-03-08T08:29:36.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.953+0000 7fbcae7fc640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 0x7fbc9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:36.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.954+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 0x7fbc9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:36.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.954+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcb40ab900 con 0x7fbca40039b0 2024-03-08T08:29:36.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.954+0000 7fbcbd25b640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 0x7fbc9c06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbcb815e740 tx=0x7fbca8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:36.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:36.958+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbcb80cfd30 con 0x7fbca40039b0 2024-03-08T08:29:37.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.136+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbcb80749c0 con 0x7fbca40039b0 2024-03-08T08:29:37.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.136+0000 7fbcae7fc640 1 -- 172.21.15.12:0/3244791489 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbcb80b6be0 con 0x7fbca40039b0 2024-03-08T08:29:37.137 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 msgr2=0x7fbc9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 0x7fbc9c06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbcb815e740 tx=0x7fbca8008040 comp rx=0 tx=0).stop 2024-03-08T08:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 msgr2=0x7fbcb80c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c4c0 tx=0x7fbcb402e070 comp rx=0 tx=0).stop 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 shutdown_connections 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbc9c06b4d0 0x7fbc9c06d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb80cc7b0 0x7fbcb80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbcb8148300 0x7fbcb8148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 --2- 172.21.15.12:0/3244791489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbca40039b0 0x7fbcb80c9b90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 >> 172.21.15.12:0/3244791489 conn(0x7fbcb806c240 msgr2=0x7fbcb80d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 shutdown_connections 2024-03-08T08:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.138+0000 7fbcbfcc0640 1 -- 172.21.15.12:0/3244791489 wait complete. 2024-03-08T08:29:37.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:37.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.253+0000 7fedde9bc640 1 Processor -- start 2024-03-08T08:29:37.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.253+0000 7fedde9bc640 1 -- start start 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedde9bc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedde9bc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 0x7fedd80cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedde9bc640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd815be80 con 0x7fedd80cd380 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedde9bc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80cdd20 con 0x7fedd815d150 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedde9bc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd806f3b0 con 0x7fedd815baa0 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedd77fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fedd80cd380 0x7fedd806abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46408/0 (socket says 172.21.15.12:46408) 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedd77fe640 1 -- 172.21.15.12:0/2128987611 learned_addr learned my addr 172.21.15.12:0/2128987611 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedd7fff640 1 --2- 172.21.15.12:0/2128987611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd806bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7feddcf59640 1 --2- 172.21.15.12:0/2128987611 >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 0x7fedd80cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedd6ffd640 1 -- 172.21.15.12:0/2128987611 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131183985 0 0) 0x7fedd815be80 con 0x7fedd80cd380 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.254+0000 7fedd6ffd640 1 -- 172.21.15.12:0/2128987611 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc0003520 con 0x7fedd80cd380 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7fedd7fff640 1 -- 172.21.15.12:0/2128987611 >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 msgr2=0x7fedd80cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7fedd7fff640 1 --2- 172.21.15.12:0/2128987611 >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 0x7fedd80cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7fedd7fff640 1 -- 172.21.15.12:0/2128987611 >> v1:172.21.15.12:6789/0 conn(0x7fedd80cd380 legacy=0x7fedd806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7fedd7fff640 1 -- 172.21.15.12:0/2128987611 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd806f690 con 0x7fedd815baa0 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7feddcf59640 1 --2- 172.21.15.12:0/2128987611 >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 0x7fedd80cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.255+0000 7fedd7fff640 1 --2- 172.21.15.12:0/2128987611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd806bf10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedc80097a0 tx=0x7fedc802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766ba9b6d2f9759d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedd6ffd640 1 -- 172.21.15.12:0/2128987611 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8035020 con 0x7fedd815baa0 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedd6ffd640 1 -- 172.21.15.12:0/2128987611 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8031830 con 0x7fedd815baa0 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedd6ffd640 1 -- 172.21.15.12:0/2128987611 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8031be0 con 0x7fedd815baa0 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 -- 172.21.15.12:0/2128987611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 msgr2=0x7fedd806bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 --2- 172.21.15.12:0/2128987611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd806bf10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedc80097a0 tx=0x7fedc802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 -- 172.21.15.12:0/2128987611 shutdown_connections 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 --2- 172.21.15.12:0/2128987611 >> v2:172.21.15.156:3300/0 conn(0x7fedd815d150 0x7fedd80cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 --2- 172.21.15.12:0/2128987611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd806bf10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 -- 172.21.15.12:0/2128987611 >> 172.21.15.12:0/2128987611 conn(0x7fedd806db00 msgr2=0x7fedd806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 -- 172.21.15.12:0/2128987611 shutdown_connections 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.256+0000 7fedde9bc640 1 -- 172.21.15.12:0/2128987611 wait complete. 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.258+0000 7fedde9bc640 1 Processor -- start 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.258+0000 7fedde9bc640 1 -- start start 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.258+0000 7fedde9bc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd80b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd815d150 0x7fedd80b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd806ff30 con 0x7fedd80cd380 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd806fc50 con 0x7fedd815d150 2024-03-08T08:29:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd806f040 con 0x7fedd815baa0 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44376/0 (socket says 172.21.15.12:44376) 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 -- 172.21.15.12:0/3156053239 learned_addr learned my addr 172.21.15.12:0/3156053239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd77fe640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd80b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7feddcf59640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd815d150 0x7fedd80b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 -- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 msgr2=0x7fedd80b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd80b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 -- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd815d150 msgr2=0x7fedd80b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd815d150 0x7fedd80b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 -- 172.21.15.12:0/3156053239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80c4f90 con 0x7fedd80cd380 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd7fff640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fedc802b910 tx=0x7fedc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8035040 con 0x7fedd80cd380 2024-03-08T08:29:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8004450 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd815a400 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8003830 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.259+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd80c5510 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.260+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd80749c0 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.261+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedc8038020 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.261+0000 7fedd4ff9640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 0x7fedc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.261+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedc80ac100 con 0x7fedd80cd380 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.261+0000 7fedd77fe640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 0x7fedc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.262+0000 7fedd77fe640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 0x7fedc406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedd80cc040 tx=0x7fedcc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.266+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedc807a9c0 con 0x7fedd80cd380 2024-03-08T08:29:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.447+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fedd80cce20 con 0x7fedd80cd380 2024-03-08T08:29:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.447+0000 7fedd4ff9640 1 -- 172.21.15.12:0/3156053239 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fedc807ea60 con 0x7fedd80cd380 2024-03-08T08:29:37.448 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 msgr2=0x7fedc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 0x7fedc406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedd80cc040 tx=0x7fedcc009040 comp rx=0 tx=0).stop 2024-03-08T08:29:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 msgr2=0x7fedd81498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fedc802b910 tx=0x7fedc80046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 shutdown_connections 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fedc406b4d0 0x7fedc406d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedd815d150 0x7fedd80b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fedd815baa0 0x7fedd80b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 --2- 172.21.15.12:0/3156053239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd80cd380 0x7fedd81498e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 >> 172.21.15.12:0/3156053239 conn(0x7fedd806db00 msgr2=0x7fedd806bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 shutdown_connections 2024-03-08T08:29:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.449+0000 7fedde9bc640 1 -- 172.21.15.12:0/3156053239 wait complete. 2024-03-08T08:29:37.461 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters' 2024-03-08T08:29:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.562+0000 7f353f5b5640 1 Processor -- start 2024-03-08T08:29:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.562+0000 7f353f5b5640 1 -- start start 2024-03-08T08:29:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353f5b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f353815ddc0 0x7f35380d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353f5b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380b8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353f5b5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353815e1a0 con 0x7f35380d0f40 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353f5b5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35380b9030 con 0x7f353815ddc0 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353f5b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35380d1d20 con 0x7f35380b7bf0 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353d351640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f353815ddc0 0x7f35380d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353cb50640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f35380d0f40 0x7f35380d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46420/0 (socket says 172.21.15.12:46420) 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353cb50640 1 -- 172.21.15.12:0/3915190304 learned_addr learned my addr 172.21.15.12:0/3915190304 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353db52640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353d351640 1 -- 172.21.15.12:0/3915190304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 msgr2=0x7f35380b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353d351640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353d351640 1 -- 172.21.15.12:0/3915190304 >> v1:172.21.15.12:6789/0 conn(0x7f35380d0f40 legacy=0x7f35380d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f353d351640 1 -- 172.21.15.12:0/3915190304 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35380b8790 con 0x7f353815ddc0 2024-03-08T08:29:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.563+0000 7f352ffff640 1 -- 172.21.15.12:0/3915190304 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060606689 0 0) 0x7f353815e1a0 con 0x7f35380d0f40 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f353d351640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380d0a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f35340097a0 tx=0x7f353402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f54a748604933c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f352ffff640 1 -- 172.21.15.12:0/3915190304 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534035020 con 0x7f353815ddc0 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f352ffff640 1 -- 172.21.15.12:0/3915190304 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3534031830 con 0x7f353815ddc0 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f352ffff640 1 -- 172.21.15.12:0/3915190304 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534031c00 con 0x7f353815ddc0 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f353f5b5640 1 -- 172.21.15.12:0/3915190304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 msgr2=0x7f35380d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.564+0000 7f353f5b5640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380d0a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f35340097a0 tx=0x7f353402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 -- 172.21.15.12:0/3915190304 shutdown_connections 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 --2- 172.21.15.12:0/3915190304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380d0a00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 -- 172.21.15.12:0/3915190304 >> 172.21.15.12:0/3915190304 conn(0x7f3538072720 msgr2=0x7f3538072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 -- 172.21.15.12:0/3915190304 shutdown_connections 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.565+0000 7f353f5b5640 1 -- 172.21.15.12:0/3915190304 wait complete. 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 Processor -- start 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 -- start start 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35380d0f40 0x7f35380b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35380b9980 con 0x7f353815ddc0 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35380b8d50 con 0x7f35380d0f40 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.566+0000 7f353f5b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35380d28c0 con 0x7f35380b7bf0 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44392/0 (socket says 172.21.15.12:44392) 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 -- 172.21.15.12:0/1759976320 learned_addr learned my addr 172.21.15.12:0/1759976320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353d351640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353cb50640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35380d0f40 0x7f35380b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 -- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 msgr2=0x7f35380ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 -- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35380d0f40 msgr2=0x7f35380b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35380d0f40 0x7f35380b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 -- 172.21.15.12:0/1759976320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353814e5b0 con 0x7f353815ddc0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353db52640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3530007e40 tx=0x7f353000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530014620 con 0x7f353815ddc0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3530019070 con 0x7f353815ddc0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353814d7f0 con 0x7f353815ddc0 2024-03-08T08:29:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530014a20 con 0x7f353815ddc0 2024-03-08T08:29:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.567+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353814ddb0 con 0x7f353815ddc0 2024-03-08T08:29:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.568+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35380749c0 con 0x7f353815ddc0 2024-03-08T08:29:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.569+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3530014bc0 con 0x7f353815ddc0 2024-03-08T08:29:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.569+0000 7f352dffb640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 0x7f351c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.569+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f353008a9a0 con 0x7f353815ddc0 2024-03-08T08:29:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.569+0000 7f353d351640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 0x7f351c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.570+0000 7f353d351640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 0x7f351c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3534036000 tx=0x7f353403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.573+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f35300598b0 con 0x7f353815ddc0 2024-03-08T08:29:37.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:37.821+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters"]} v 0) v1 -- 0x7f35380b84d0 con 0x7f353815ddc0 2024-03-08T08:29:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.337+0000 7f352dffb640 1 -- 172.21.15.12:0/1759976320 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters"]}]=0 v45) v1 ==== 167+0+0 (secure 0 0 0) 0x7f35300591b0 con 0x7f353815ddc0 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 msgr2=0x7f351c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 0x7f351c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3534036000 tx=0x7f353403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 msgr2=0x7f35380b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3530007e40 tx=0x7f353000e4b0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 shutdown_connections 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f351c06b4d0 0x7f351c06d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353815ddc0 0x7f35380b4390 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35380d0f40 0x7f35380b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 --2- 172.21.15.12:0/1759976320 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35380b7bf0 0x7f35380ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 >> 172.21.15.12:0/1759976320 conn(0x7f3538072720 msgr2=0x7f353806c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 shutdown_connections 2024-03-08T08:29:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.340+0000 7f353f5b5640 1 -- 172.21.15.12:0/1759976320 wait complete. 2024-03-08T08:29:38.353 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-08T08:29:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.460+0000 7f5941fbc640 1 Processor -- start 2024-03-08T08:29:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.460+0000 7f5941fbc640 1 -- start start 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f5941fbc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f593c0b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f5941fbc640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0b7250 con 0x7f593c0b9760 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f5941fbc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0b7520 con 0x7f593c15dbc0 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f5941fbc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b8f70 con 0x7f593c0b85c0 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f593affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f593c0b9760 0x7f593c0d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46432/0 (socket says 172.21.15.12:46432) 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f593affd640 1 -- 172.21.15.12:0/1603888641 learned_addr learned my addr 172.21.15.12:0/1603888641 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334018315 0 0) 0x7f593c0b7250 con 0x7f593c0b9760 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924003520 con 0x7f593c0b9760 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.461+0000 7f593bfff640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f593c0b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588628450 0 0) 0x7f5924003520 con 0x7f593c0b9760 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 msgr2=0x7f593c0b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f593c0b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 >> v1:172.21.15.156:6789/0 conn(0x7f593c15dbc0 legacy=0x7f593c0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0b7800 con 0x7f593c0b9760 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f5930001670 con 0x7f593c0b9760 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 >> v1:172.21.15.12:6789/0 conn(0x7f593c0b9760 legacy=0x7f593c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f5924003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5924003f80 0x7f5924004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5924004d40 0x7f59240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b8f70 con 0x7f5924004d40 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592400bb60 con 0x7f5924003f80 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592400bd60 con 0x7f593c0b85c0 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5924004d40 0x7f59240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593affd640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5924003f80 0x7f5924004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f5924003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 -- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 msgr2=0x7f5924003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f5924003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 -- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5924003f80 msgr2=0x7f5924004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5924003f80 0x7f5924004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593bfff640 1 -- 172.21.15.12:0/1603888641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924003520 con 0x7f5924004d40 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593affd640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5924003f80 0x7f5924004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.462+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0b85c0 0x7f5924003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.463+0000 7f593bfff640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5924004d40 0x7f59240051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5934009f90 tx=0x7f5924005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92c9139b53521d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.463+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934039070 con 0x7f5924004d40 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.463+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593402cbd0 con 0x7f5924004d40 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.463+0000 7f593a7fc640 1 -- 172.21.15.12:0/1603888641 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934009c10 con 0x7f5924004d40 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 -- 172.21.15.12:0/1603888641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5924004d40 msgr2=0x7f59240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1603888641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5924004d40 0x7f59240051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5934009f90 tx=0x7f5924005830 comp rx=0 tx=0).stop 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f593bfff640 1 -- 172.21.15.12:0/1603888641 reap_dead start 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 -- 172.21.15.12:0/1603888641 shutdown_connections 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 -- 172.21.15.12:0/1603888641 >> 172.21.15.12:0/1603888641 conn(0x7f593c072720 msgr2=0x7f593c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 -- 172.21.15.12:0/1603888641 shutdown_connections 2024-03-08T08:29:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.464+0000 7f5941fbc640 1 -- 172.21.15.12:0/1603888641 wait complete. 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.465+0000 7f5941fbc640 1 Processor -- start 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.465+0000 7f5941fbc640 1 -- start start 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 0x7f593c0aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 0x7f593c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b80a0 con 0x7f593c15ff60 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b7dc0 con 0x7f593c0b1f30 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d2220 con 0x7f593c0aaa00 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44422/0 (socket says 172.21.15.12:44422) 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 -- 172.21.15.12:0/1796473881 learned_addr learned my addr 172.21.15.12:0/1796473881 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 0x7f593c0aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593affd640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 0x7f593c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 -- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 msgr2=0x7f593c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 0x7f593c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 -- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 msgr2=0x7f593c0aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 0x7f593c0aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 -- 172.21.15.12:0/1796473881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c160780 con 0x7f593c15ff60 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 0x7f593c0aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593affd640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 0x7f593c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f593bfff640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5934037040 tx=0x7f5934009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934039030 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593403a040 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c14ae70 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934009c10 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.466+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c160410 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.467+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c0749c0 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.468+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5934045030 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.468+0000 7f5923fff640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 0x7f592806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.468+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59340abe90 con 0x7f593c15ff60 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.468+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 0x7f592806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.468+0000 7f593b7fe640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 0x7f592806d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f593c15b8b0 tx=0x7f592c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.472+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f593404c060 con 0x7f593c15ff60 2024-03-08T08:29:38.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.655+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f593c0aade0 con 0x7f593c15ff60 2024-03-08T08:29:38.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.658+0000 7f5923fff640 1 -- 172.21.15.12:0/1796473881 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f593407ada0 con 0x7f593c15ff60 2024-03-08T08:29:38.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 msgr2=0x7f592806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 0x7f592806d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f593c15b8b0 tx=0x7f592c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 msgr2=0x7f593c14cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5934037040 tx=0x7f5934009750 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 shutdown_connections 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f592806b500 0x7f592806d9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f593c15ff60 0x7f593c14cd00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f593c0aaa00 0x7f593c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 --2- 172.21.15.12:0/1796473881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f593c0b1f30 0x7f593c0aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 >> 172.21.15.12:0/1796473881 conn(0x7f593c072720 msgr2=0x7f593c0cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 shutdown_connections 2024-03-08T08:29:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.660+0000 7f5941fbc640 1 -- 172.21.15.12:0/1796473881 wait complete. 2024-03-08T08:29:38.672 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-08T08:29:38.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-08T08:29:38.715 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-08T08:29:38.760 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-08T08:29:38.760 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:38.761 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:38.775 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:38.775 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:38.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:38.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.970+0000 7f3bd3cdf640 1 Processor -- start 2024-03-08T08:29:38.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.970+0000 7f3bd3cdf640 1 -- start start 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd3cdf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0be0 0x7f3bcc15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd3cdf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd3cdf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc0cd3b0 con 0x7f3bcc06b110 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd3cdf640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bcc0cd690 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd3cdf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0cd970 con 0x7f3bcc0b0be0 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd127a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd127a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49508/0 (socket says 172.21.15.12:49508) 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.971+0000 7f3bd127a640 1 -- 172.21.15.12:0/2373255176 learned_addr learned my addr 172.21.15.12:0/2373255176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd1a7b640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0be0 0x7f3bcc15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd127a640 1 -- 172.21.15.12:0/2373255176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0be0 msgr2=0x7f3bcc15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd127a640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0be0 0x7f3bcc15c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd127a640 1 -- 172.21.15.12:0/2373255176 >> v1:172.21.15.156:6789/0 conn(0x7f3bcc06b110 legacy=0x7f3bcc0d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd127a640 1 -- 172.21.15.12:0/2373255176 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0cdc50 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.972+0000 7f3bd127a640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009870 tx=0x7f3bc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b78e6480ecb2fb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd0a79640 1 -- 172.21.15.12:0/2373255176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0034070 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd0a79640 1 -- 172.21.15.12:0/2373255176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc0030840 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd0a79640 1 -- 172.21.15.12:0/2373255176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0030c10 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/2373255176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc0b0fc0 msgr2=0x7f3bcc06abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009870 tx=0x7f3bc002b980 comp rx=0 tx=0).stop 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/2373255176 shutdown_connections 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc0b0fc0 0x7f3bcc06abd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/2373255176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0be0 0x7f3bcc15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/2373255176 >> 172.21.15.12:0/2373255176 conn(0x7f3bcc099a40 msgr2=0x7f3bcc099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/2373255176 shutdown_connections 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.973+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/2373255176 wait complete. 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.974+0000 7f3bd3cdf640 1 Processor -- start 2024-03-08T08:29:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.974+0000 7f3bd3cdf640 1 -- start start 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bcc0b0be0 0x7f3bcc1e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0fc0 0x7f3bcc1442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc15cee0 con 0x7f3bcc06b110 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b18c0 con 0x7f3bcc0b0be0 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd3cdf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0ccf60 con 0x7f3bcc0b0fc0 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd1a7b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd1a7b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49518/0 (socket says 172.21.15.12:49518) 2024-03-08T08:29:38.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd1a7b640 1 -- 172.21.15.12:0/3485952299 learned_addr learned my addr 172.21.15.12:0/3485952299 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd127a640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bcc0b0be0 0x7f3bcc1e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.975+0000 7f3bd227c640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0fc0 0x7f3bcc1442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 -- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0fc0 msgr2=0x7f3bcc1442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0fc0 0x7f3bcc1442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 -- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bcc0b0be0 msgr2=0x7f3bcc1e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bcc0b0be0 0x7f3bcc1e6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 -- 172.21.15.12:0/3485952299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0b50c0 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd1a7b640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400d800 tx=0x7f3bc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400b6c0 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc4019070 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc142230 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400ba80 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.976+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc1427f0 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.977+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc0749c0 con 0x7f3bcc06b110 2024-03-08T08:29:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.977+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bc400bc20 con 0x7f3bcc06b110 2024-03-08T08:29:38.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.978+0000 7f3bca7fc640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 0x7f3bb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:38.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.978+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bc408a720 con 0x7f3bcc06b110 2024-03-08T08:29:38.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.978+0000 7f3bd127a640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 0x7f3bb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:38.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.978+0000 7f3bd127a640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 0x7f3bb806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0036000 tx=0x7f3bc003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:38.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:38.982+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3bc4059510 con 0x7f3bcc06b110 2024-03-08T08:29:39.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.164+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bcc15ba60 con 0x7f3bcc06b110 2024-03-08T08:29:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.165+0000 7f3bca7fc640 1 -- 172.21.15.12:0/3485952299 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f3bc4058e10 con 0x7f3bcc06b110 2024-03-08T08:29:39.166 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:39.167 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:39.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 msgr2=0x7f3bb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 0x7f3bb806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0036000 tx=0x7f3bc003b040 comp rx=0 tx=0).stop 2024-03-08T08:29:39.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 msgr2=0x7f3bcc1e6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400d800 tx=0x7f3bc400dcc0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 shutdown_connections 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3bb806b4d0 0x7f3bb806d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3bcc0b0fc0 0x7f3bcc1442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bcc0b0be0 0x7f3bcc1e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 --2- 172.21.15.12:0/3485952299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bcc06b110 0x7f3bcc1e6600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 >> 172.21.15.12:0/3485952299 conn(0x7f3bcc099a40 msgr2=0x7f3bcc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 shutdown_connections 2024-03-08T08:29:39.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.167+0000 7f3bd3cdf640 1 -- 172.21.15.12:0/3485952299 wait complete. 2024-03-08T08:29:39.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:39.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:39.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.282+0000 7f0172dae640 1 Processor -- start 2024-03-08T08:29:39.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.282+0000 7f0172dae640 1 -- start start 2024-03-08T08:29:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0172dae640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f016c099b90 0x7f016c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0172dae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 0x7f016c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0172dae640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016c0d0290 con 0x7f016c15e0d0 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0172dae640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016c0d0570 con 0x7f016c099b90 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0172dae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0d0850 con 0x7f016c06bef0 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f016c099b90 0x7f016c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f016c099b90 0x7f016c0cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49530/0 (socket says 172.21.15.12:49530) 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 -- 172.21.15.12:0/72326482 learned_addr learned my addr 172.21.15.12:0/72326482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f017134b640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 0x7f016c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 -- 172.21.15.12:0/72326482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 msgr2=0x7f016c0cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 0x7f016c0cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.283+0000 7f0170b4a640 1 -- 172.21.15.12:0/72326482 >> v1:172.21.15.156:6789/0 conn(0x7f016c15e0d0 legacy=0x7f016c099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f0170b4a640 1 -- 172.21.15.12:0/72326482 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0d0b30 con 0x7f016c099b90 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f017134b640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 0x7f016c0cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f0170b4a640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0cfc40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa331d2d7ab88a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f016b7fe640 1 -- 172.21.15.12:0/72326482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c035020 con 0x7f016c099b90 2024-03-08T08:29:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f016b7fe640 1 -- 172.21.15.12:0/72326482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c031640 con 0x7f016c099b90 2024-03-08T08:29:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.284+0000 7f016b7fe640 1 -- 172.21.15.12:0/72326482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c02d040 con 0x7f016c099b90 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 -- 172.21.15.12:0/72326482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 msgr2=0x7f016c0cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0cfc40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c02b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 -- 172.21.15.12:0/72326482 shutdown_connections 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c06bef0 0x7f016c0cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 --2- 172.21.15.12:0/72326482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0cfc40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 -- 172.21.15.12:0/72326482 >> 172.21.15.12:0/72326482 conn(0x7f016c0b3460 msgr2=0x7f016c0b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 -- 172.21.15.12:0/72326482 shutdown_connections 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.285+0000 7f0172dae640 1 -- 172.21.15.12:0/72326482 wait complete. 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.286+0000 7f0172dae640 1 Processor -- start 2024-03-08T08:29:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.286+0000 7f0172dae640 1 -- start start 2024-03-08T08:29:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 0x7f016c0b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 0x7f016c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c09b310 con 0x7f016c099b90 2024-03-08T08:29:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c09b190 con 0x7f016c06bef0 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0172dae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0d1330 con 0x7f016c15b810 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49532/0 (socket says 172.21.15.12:49532) 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 -- 172.21.15.12:0/4105933083 learned_addr learned my addr 172.21.15.12:0/4105933083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f017134b640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 0x7f016c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0170b4a640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 0x7f016c0b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 -- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 msgr2=0x7f016c0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 0x7f016c0b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 -- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 msgr2=0x7f016c0b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 0x7f016c0b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f016bfff640 1 -- 172.21.15.12:0/4105933083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0b6d70 con 0x7f016c099b90 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f0170b4a640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 0x7f016c0b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.287+0000 7f017134b640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 0x7f016c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f016bfff640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f016000b3f0 tx=0x7f016000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160015020 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0160011830 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c0b7050 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160011c20 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.288+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c0b7610 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.289+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c0749c0 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.289+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0160018020 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.290+0000 7f01697fa640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.290+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0160089e50 con 0x7f016c099b90 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.290+0000 7f0170b4a640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.290+0000 7f0170b4a640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 0x7f015806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f015c036000 tx=0x7f015c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.294+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f016001f030 con 0x7f016c099b90 2024-03-08T08:29:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.472+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f016c0cd840 con 0x7f016c099b90 2024-03-08T08:29:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.472+0000 7f01697fa640 1 -- 172.21.15.12:0/4105933083 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f016008b0a0 con 0x7f016c099b90 2024-03-08T08:29:39.474 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 msgr2=0x7f015806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 0x7f015806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f015c036000 tx=0x7f015c03c040 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 msgr2=0x7f016c0afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f016000b3f0 tx=0x7f016000b8b0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 shutdown_connections 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f016c15b810 0x7f016c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f016c099b90 0x7f016c0afd60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 --2- 172.21.15.12:0/4105933083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f016c06bef0 0x7f016c0b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 >> 172.21.15.12:0/4105933083 conn(0x7f016c0b3460 msgr2=0x7f016c15c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 shutdown_connections 2024-03-08T08:29:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.474+0000 7f0172dae640 1 -- 172.21.15.12:0/4105933083 wait complete. 2024-03-08T08:29:39.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:39.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.589+0000 7f0035edf640 1 Processor -- start 2024-03-08T08:29:39.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.589+0000 7f0035edf640 1 -- start start 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f0035edf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 0x7f00300cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f0035edf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0030099b90 0x7f003009a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f0035edf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00300cdf60 con 0x7f003015bb90 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f0035edf640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300cd590 con 0x7f0030099b90 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f0035edf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300cf700 con 0x7f00300cc390 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f002ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0030099b90 0x7f003009a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f002ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0030099b90 0x7f003009a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49538/0 (socket says 172.21.15.12:49538) 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.590+0000 7f002ffff640 1 -- 172.21.15.12:0/39653325 learned_addr learned my addr 172.21.15.12:0/39653325 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002effd640 1 --2- 172.21.15.12:0/39653325 >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 0x7f00300cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002ffff640 1 -- 172.21.15.12:0/39653325 >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 msgr2=0x7f00300cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002ffff640 1 --2- 172.21.15.12:0/39653325 >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 0x7f00300cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002ffff640 1 -- 172.21.15.12:0/39653325 >> v1:172.21.15.156:6789/0 conn(0x7f003015bb90 legacy=0x7f00300d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002ffff640 1 -- 172.21.15.12:0/39653325 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00300cc980 con 0x7f0030099b90 2024-03-08T08:29:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002effd640 1 --2- 172.21.15.12:0/39653325 >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 0x7f00300cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:39.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.591+0000 7f002ffff640 1 --2- 172.21.15.12:0/39653325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0030099b90 0x7f003009a020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f003009f8c0 tx=0x7f002802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964287d77ac0f077 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f002e7fc640 1 -- 172.21.15.12:0/39653325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002802ca80 con 0x7f0030099b90 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f002e7fc640 1 -- 172.21.15.12:0/39653325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002802cc20 con 0x7f0030099b90 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f002e7fc640 1 -- 172.21.15.12:0/39653325 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280343f0 con 0x7f0030099b90 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 -- 172.21.15.12:0/39653325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0030099b90 msgr2=0x7f003009a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 --2- 172.21.15.12:0/39653325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0030099b90 0x7f003009a020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f003009f8c0 tx=0x7f002802bee0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 -- 172.21.15.12:0/39653325 shutdown_connections 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 --2- 172.21.15.12:0/39653325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0030099b90 0x7f003009a020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 --2- 172.21.15.12:0/39653325 >> v2:172.21.15.156:3300/0 conn(0x7f00300cc390 0x7f00300cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 -- 172.21.15.12:0/39653325 >> 172.21.15.12:0/39653325 conn(0x7f00300b3370 msgr2=0x7f00300b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 -- 172.21.15.12:0/39653325 shutdown_connections 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.592+0000 7f0035edf640 1 -- 172.21.15.12:0/39653325 wait complete. 2024-03-08T08:29:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.593+0000 7f0035edf640 1 Processor -- start 2024-03-08T08:29:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 -- start start 2024-03-08T08:29:39.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0030099b90 0x7f0030149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f003015bb90 0x7f00300b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300cd220 con 0x7f00300cc390 2024-03-08T08:29:39.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300ccf40 con 0x7f003015bb90 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f0035edf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003015bf70 con 0x7f0030099b90 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f002effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f002effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49546/0 (socket says 172.21.15.12:49546) 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.594+0000 7f002effd640 1 -- 172.21.15.12:0/749305219 learned_addr learned my addr 172.21.15.12:0/749305219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002f7fe640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0030099b90 0x7f0030149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002ffff640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f003015bb90 0x7f00300b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 -- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0030099b90 msgr2=0x7f0030149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0030099b90 0x7f0030149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 -- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f003015bb90 msgr2=0x7f00300b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f003015bb90 0x7f00300b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 -- 172.21.15.12:0/749305219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003015eeb0 con 0x7f00300cc390 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f002effd640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f002400c7f0 tx=0x7f002400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024015070 con 0x7f00300cc390 2024-03-08T08:29:39.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024007a30 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024007e00 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00300b6b60 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.595+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00300b7120 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.596+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00300d1210 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.596+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0024018070 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.597+0000 7f0017fff640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 0x7f001c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.597+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f002408a170 con 0x7f00300cc390 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.597+0000 7f002f7fe640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 0x7f001c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.597+0000 7f002f7fe640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 0x7f001c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0020003d90 tx=0x7f0020009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.601+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0024059080 con 0x7f00300cc390 2024-03-08T08:29:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.782+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00300b28e0 con 0x7f00300cc390 2024-03-08T08:29:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.782+0000 7f0017fff640 1 -- 172.21.15.12:0/749305219 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0024058980 con 0x7f00300cc390 2024-03-08T08:29:39.783 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 msgr2=0x7f001c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 0x7f001c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0020003d90 tx=0x7f0020009040 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 msgr2=0x7f00300b5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f002400c7f0 tx=0x7f002400cbf0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 shutdown_connections 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f001c06b4d0 0x7f001c06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f003015bb90 0x7f00300b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00300cc390 0x7f00300b5d90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 --2- 172.21.15.12:0/749305219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0030099b90 0x7f0030149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 >> 172.21.15.12:0/749305219 conn(0x7f00300b3370 msgr2=0x7f003006f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.784+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 shutdown_connections 2024-03-08T08:29:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.785+0000 7f0035edf640 1 -- 172.21.15.12:0/749305219 wait complete. 2024-03-08T08:29:39.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.900+0000 7ff345e83640 1 Processor -- start 2024-03-08T08:29:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.900+0000 7ff345e83640 1 -- start start 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.900+0000 7ff345e83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff345e83640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff3400b7080 0x7ff3400d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff345e83640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400b7520 con 0x7ff3400d2050 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff345e83640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3400b7800 con 0x7ff3400b7080 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff345e83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400b7ae0 con 0x7ff34015dbc0 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff3400b7080 0x7ff3400d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:41452/0 (socket says 172.21.15.12:41452) 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff3400b7080 0x7ff3400d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:38802/0 (socket says 172.21.15.12:38802) 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33f7fe640 1 -- 172.21.15.12:0/198778493 learned_addr learned my addr 172.21.15.12:0/198778493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33ffff640 1 --1- 172.21.15.12:0/198778493 >> v1:172.21.15.156:6789/0 conn(0x7ff3400d2050 0x7ff3400b85c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:52884/0 (socket says 172.21.15.12:52884) 2024-03-08T08:29:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 -- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 msgr2=0x7ff3400b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 -- 172.21.15.12:0/198778493 >> v1:172.21.15.156:6789/0 conn(0x7ff3400d2050 legacy=0x7ff3400b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33effd640 1 -- 172.21.15.12:0/198778493 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400b7dc0 con 0x7ff3400b7080 2024-03-08T08:29:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.901+0000 7ff33f7fe640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.902+0000 7ff33effd640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3400b7080 0x7ff3400d1b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3340097a0 tx=0x7ff33402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212b9e28e88b1687 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.902+0000 7ff33e7fc640 1 -- 172.21.15.12:0/198778493 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334034070 con 0x7ff3400b7080 2024-03-08T08:29:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.902+0000 7ff33e7fc640 1 -- 172.21.15.12:0/198778493 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334030610 con 0x7ff3400b7080 2024-03-08T08:29:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.902+0000 7ff33e7fc640 1 -- 172.21.15.12:0/198778493 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3340309e0 con 0x7ff3400b7080 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 -- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3400b7080 msgr2=0x7ff3400d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3400b7080 0x7ff3400d1b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3340097a0 tx=0x7ff33402b960 comp rx=0 tx=0).stop 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 -- 172.21.15.12:0/198778493 shutdown_connections 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3400b7080 0x7ff3400d1b10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 --2- 172.21.15.12:0/198778493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff34015dbc0 0x7ff3400b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 -- 172.21.15.12:0/198778493 >> 172.21.15.12:0/198778493 conn(0x7ff340072720 msgr2=0x7ff340072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 -- 172.21.15.12:0/198778493 shutdown_connections 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.903+0000 7ff345e83640 1 -- 172.21.15.12:0/198778493 wait complete. 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.904+0000 7ff345e83640 1 Processor -- start 2024-03-08T08:29:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.904+0000 7ff345e83640 1 -- start start 2024-03-08T08:29:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3400d2050 0x7ff3400cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff34015dbc0 0x7ff3400b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400b8cd0 con 0x7ff3400b7080 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34015dfa0 con 0x7ff34015dbc0 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff345e83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400b8e50 con 0x7ff3400d2050 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49562/0 (socket says 172.21.15.12:49562) 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 -- 172.21.15.12:0/1487465160 learned_addr learned my addr 172.21.15.12:0/1487465160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33ffff640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff34015dbc0 0x7ff3400b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33effd640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3400d2050 0x7ff3400cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 -- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3400d2050 msgr2=0x7ff3400cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3400d2050 0x7ff3400cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 -- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff34015dbc0 msgr2=0x7ff3400b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff34015dbc0 0x7ff3400b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.905+0000 7ff33f7fe640 1 -- 172.21.15.12:0/1487465160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400cb0e0 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff33f7fe640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff33000e9b0 tx=0x7ff33000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330017070 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3400cb3b0 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000c9a0 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000cd50 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.906+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3400cb970 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.907+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3400749c0 con 0x7ff3400b7080 2024-03-08T08:29:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.907+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff33001a070 con 0x7ff3400b7080 2024-03-08T08:29:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.908+0000 7ff327fff640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 0x7ff32c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.908+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff33008a050 con 0x7ff3400b7080 2024-03-08T08:29:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.908+0000 7ff33effd640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 0x7ff32c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.908+0000 7ff33effd640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 0x7ff32c06d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff334036000 tx=0x7ff33403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:39.912+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff330058ee0 con 0x7ff3400b7080 2024-03-08T08:29:40.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.094+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3400c9dc0 con 0x7ff3400b7080 2024-03-08T08:29:40.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.094+0000 7ff327fff640 1 -- 172.21.15.12:0/1487465160 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3300587e0 con 0x7ff3400b7080 2024-03-08T08:29:40.095 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 msgr2=0x7ff32c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 0x7ff32c06d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff334036000 tx=0x7ff33403b040 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 msgr2=0x7ff3400caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff33000e9b0 tx=0x7ff33000ee70 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 shutdown_connections 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff32c06b500 0x7ff32c06d9c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff34015dbc0 0x7ff3400b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3400d2050 0x7ff3400cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 --2- 172.21.15.12:0/1487465160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3400b7080 0x7ff3400caba0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.096+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 >> 172.21.15.12:0/1487465160 conn(0x7ff340072720 msgr2=0x7ff3400d2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.097+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 shutdown_connections 2024-03-08T08:29:40.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.097+0000 7ff345e83640 1 -- 172.21.15.12:0/1487465160 wait complete. 2024-03-08T08:29:40.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.210+0000 7f58b6046640 1 Processor -- start 2024-03-08T08:29:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.210+0000 7f58b6046640 1 -- start start 2024-03-08T08:29:40.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58b6046640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58b015dbd0 0x7f58b00d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58b6046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b006c870 0x7f58b00b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58b6046640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b00d1840 con 0x7f58b00b9440 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58b6046640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b00d25f0 con 0x7f58b015dbd0 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58b6046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00d28d0 con 0x7f58b006c870 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58af7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58b015dbd0 0x7f58b00d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58aeffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f58b00b9440 0x7f58b006c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44404/0 (socket says 172.21.15.12:44404) 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58aeffd640 1 -- 172.21.15.12:0/131840778 learned_addr learned my addr 172.21.15.12:0/131840778 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58affff640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b006c870 0x7f58b00b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58af7fe640 1 -- 172.21.15.12:0/131840778 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b006c870 msgr2=0x7f58b00b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58af7fe640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b006c870 0x7f58b00b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58af7fe640 1 -- 172.21.15.12:0/131840778 >> v1:172.21.15.12:6789/0 conn(0x7f58b00b9440 legacy=0x7f58b006c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58ae7fc640 1 -- 172.21.15.12:0/131840778 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395124524 0 0) 0x7f58b00d1840 con 0x7f58b00b9440 2024-03-08T08:29:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.211+0000 7f58af7fe640 1 -- 172.21.15.12:0/131840778 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b00d2bb0 con 0x7f58b015dbd0 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.212+0000 7f58af7fe640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b015dbd0 0x7f58b00d1f80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd72cbd74181a66f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.212+0000 7f58ae7fc640 1 -- 172.21.15.12:0/131840778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0035020 con 0x7f58b015dbd0 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.212+0000 7f58ae7fc640 1 -- 172.21.15.12:0/131840778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a0031830 con 0x7f58b015dbd0 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.212+0000 7f58ae7fc640 1 -- 172.21.15.12:0/131840778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0031c00 con 0x7f58b015dbd0 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 -- 172.21.15.12:0/131840778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b015dbd0 msgr2=0x7f58b00d1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b015dbd0 0x7f58b00d1f80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 -- 172.21.15.12:0/131840778 shutdown_connections 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b006c870 0x7f58b00b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 --2- 172.21.15.12:0/131840778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b015dbd0 0x7f58b00d1f80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 -- 172.21.15.12:0/131840778 >> 172.21.15.12:0/131840778 conn(0x7f58b0072930 msgr2=0x7f58b00d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 -- 172.21.15.12:0/131840778 shutdown_connections 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.213+0000 7f58b6046640 1 -- 172.21.15.12:0/131840778 wait complete. 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.214+0000 7f58b6046640 1 Processor -- start 2024-03-08T08:29:40.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.214+0000 7f58b6046640 1 -- start start 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58b00b9440 0x7f58b00ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b015dbd0 0x7f58b00c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00d3690 con 0x7f58b006c870 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00d3170 con 0x7f58b00b9440 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58b6046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00d3810 con 0x7f58b015dbd0 2024-03-08T08:29:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49582/0 (socket says 172.21.15.12:49582) 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 -- 172.21.15.12:0/11815287 learned_addr learned my addr 172.21.15.12:0/11815287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 -- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b015dbd0 msgr2=0x7f58b00c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b015dbd0 0x7f58b00c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 -- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58b00b9440 msgr2=0x7f58b00ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58b00b9440 0x7f58b00ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.215+0000 7f58af7fe640 1 -- 172.21.15.12:0/11815287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b01f76c0 con 0x7f58b006c870 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f58af7fe640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034040 tx=0x7f58a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0035040 con 0x7f58b006c870 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a0004230 con 0x7f58b006c870 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b01f7980 con 0x7f58b006c870 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a003a640 con 0x7f58b006c870 2024-03-08T08:29:40.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.216+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b01f7f40 con 0x7f58b006c870 2024-03-08T08:29:40.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.217+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b00749c0 con 0x7f58b006c870 2024-03-08T08:29:40.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.217+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58a0038020 con 0x7f58b006c870 2024-03-08T08:29:40.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.218+0000 7f5897fff640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.218+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58a00abe70 con 0x7f58b006c870 2024-03-08T08:29:40.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.218+0000 7f58aeffd640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.218+0000 7f58aeffd640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f58a4007fd0 tx=0x7f58a4007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.222+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f58a007acf0 con 0x7f58b006c870 2024-03-08T08:29:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.415+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f58b015e120 con 0x7f58b006c870 2024-03-08T08:29:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.415+0000 7f5897fff640 1 -- 172.21.15.12:0/11815287 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f58a007a5f0 con 0x7f58b006c870 2024-03-08T08:29:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 msgr2=0x7f589c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f58a4007fd0 tx=0x7f58a4007920 comp rx=0 tx=0).stop 2024-03-08T08:29:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 msgr2=0x7f58b0156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034040 tx=0x7f58a00046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 shutdown_connections 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f58b015dbd0 0x7f58b00c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58b00b9440 0x7f58b00ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 --2- 172.21.15.12:0/11815287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b006c870 0x7f58b0156950 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 >> 172.21.15.12:0/11815287 conn(0x7f58b0072930 msgr2=0x7f58b00b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 shutdown_connections 2024-03-08T08:29:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.418+0000 7f58b6046640 1 -- 172.21.15.12:0/11815287 wait complete. 2024-03-08T08:29:40.430 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:40.430 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:40.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:40.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.585+0000 7f0c32125640 1 Processor -- start 2024-03-08T08:29:40.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.585+0000 7f0c32125640 1 -- start start 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c32125640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0c2c15ddc0 0x7f0c2c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c32125640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c15e1a0 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c32125640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c0b8790 con 0x7f0c2c0b7bf0 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c32125640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2c0b9030 con 0x7f0c2c15ddc0 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c2c0d0f40 0x7f0c2c0d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44416/0 (socket says 172.21.15.12:44416) 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2affd640 1 -- 172.21.15.12:0/281804085 learned_addr learned my addr 172.21.15.12:0/281804085 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/281804085 >> v2:172.21.15.156:3300/0 conn(0x7f0c2c15ddc0 0x7f0c2c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502135760 0 0) 0x7f0c2c15e1a0 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c14003520 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3959350800 0 0) 0x7f0c14003520 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 >> v2:172.21.15.156:3300/0 conn(0x7f0c2c15ddc0 msgr2=0x7f0c2c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 --2- 172.21.15.12:0/281804085 >> v2:172.21.15.156:3300/0 conn(0x7f0c2c15ddc0 0x7f0c2c0d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 >> v1:172.21.15.156:6789/0 conn(0x7f0c2c0b7bf0 legacy=0x7f0c2c0d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.586+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0b8a70 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f0c20001670 con 0x7f0c2c0d0f40 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 >> v1:172.21.15.12:6789/0 conn(0x7f0c2c0d0f40 legacy=0x7f0c2c0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c140039b0 0x7f0c14003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c140043b0 0x7f0c14004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1400bfb0 con 0x7f0c14005080 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1400c160 con 0x7f0c140043b0 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1400c360 con 0x7f0c140039b0 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2affd640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c140043b0 0x7f0c14004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c140039b0 0x7f0c14003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 -- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c140039b0 msgr2=0x7f0c14003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c140039b0 0x7f0c14003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 -- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c140043b0 msgr2=0x7f0c14004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c140043b0 0x7f0c14004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2bfff640 1 -- 172.21.15.12:0/281804085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c14003520 con 0x7f0c14005080 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.587+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c140039b0 0x7f0c14003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c2bfff640 1 -- 172.21.15.12:0/281804085 reap_dead start 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c2bfff640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c1400b9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0c24002010 tx=0x7f0c14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b393d60292a573 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24001be0 con 0x7f0c14005080 2024-03-08T08:29:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c24003b70 con 0x7f0c14005080 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c2a7fc640 1 -- 172.21.15.12:0/281804085 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24005110 con 0x7f0c14005080 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 -- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 msgr2=0x7f0c1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c1400b9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0c24002010 tx=0x7f0c14005830 comp rx=0 tx=0).stop 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 -- 172.21.15.12:0/281804085 shutdown_connections 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 --2- 172.21.15.12:0/281804085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c1400b9c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 -- 172.21.15.12:0/281804085 >> 172.21.15.12:0/281804085 conn(0x7f0c2c072720 msgr2=0x7f0c2c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 -- 172.21.15.12:0/281804085 shutdown_connections 2024-03-08T08:29:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.588+0000 7f0c32125640 1 -- 172.21.15.12:0/281804085 wait complete. 2024-03-08T08:29:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 Processor -- start 2024-03-08T08:29:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 -- start start 2024-03-08T08:29:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c2c0b7550 0x7f0c2c1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c2c0aace0 0x7f0c2c0ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0b9980 con 0x7f0c14005080 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0d1d20 con 0x7f0c2c0b7550 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.590+0000 7f0c32125640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0d0a00 con 0x7f0c2c0aace0 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49588/0 (socket says 172.21.15.12:49588) 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 -- 172.21.15.12:0/1136537986 learned_addr learned my addr 172.21.15.12:0/1136537986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2affd640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c2c0b7550 0x7f0c2c1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2bfff640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c2c0aace0 0x7f0c2c0ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 -- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c2c0aace0 msgr2=0x7f0c2c0ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c2c0aace0 0x7f0c2c0ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 -- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c2c0b7550 msgr2=0x7f0c2c1f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c2c0b7550 0x7f0c2c1f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 -- 172.21.15.12:0/1136537986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0aa550 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c2b7fe640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c011540 tx=0x7f0c1c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c015870 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c015a10 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c015dc0 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c1f3870 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.591+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c0aa7c0 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.592+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c0749c0 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.593+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c1c01d400 con 0x7f0c14005080 2024-03-08T08:29:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.593+0000 7f0c13fff640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 0x7f0c1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.593+0000 7f0c2affd640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 0x7f0c1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.593+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c1c08a9e0 con 0x7f0c14005080 2024-03-08T08:29:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.594+0000 7f0c2affd640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 0x7f0c1806d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c161010 tx=0x7f0c20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.598+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0c1c019030 con 0x7f0c14005080 2024-03-08T08:29:40.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.779+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c2c0b8150 con 0x7f0c14005080 2024-03-08T08:29:40.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.780+0000 7f0c13fff640 1 -- 172.21.15.12:0/1136537986 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f0c1c059860 con 0x7f0c14005080 2024-03-08T08:29:40.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:40.782 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 msgr2=0x7f0c1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 0x7f0c1806d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c161010 tx=0x7f0c20009040 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 msgr2=0x7f0c2c0ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c011540 tx=0x7f0c1c011a00 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 shutdown_connections 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0c1806b500 0x7f0c1806d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c2c0aace0 0x7f0c2c0ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c2c0b7550 0x7f0c2c1f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 --2- 172.21.15.12:0/1136537986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14005080 0x7f0c2c0ae100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 >> 172.21.15.12:0/1136537986 conn(0x7f0c2c072720 msgr2=0x7f0c2c0d3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.782+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 shutdown_connections 2024-03-08T08:29:40.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.783+0000 7f0c32125640 1 -- 172.21.15.12:0/1136537986 wait complete. 2024-03-08T08:29:40.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:40.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:40.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.897+0000 7fca83a4e640 1 Processor -- start 2024-03-08T08:29:40.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.897+0000 7fca83a4e640 1 -- start start 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca83a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 0x7fca7c0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca83a4e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fca7c099b90 0x7fca7c09a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca83a4e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0cdf60 con 0x7fca7c15bb90 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca83a4e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0cd590 con 0x7fca7c099b90 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca83a4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cf700 con 0x7fca7c0cc390 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca817ea640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fca7c15bb90 0x7fca7c0d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44424/0 (socket says 172.21.15.12:44424) 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca817ea640 1 -- 172.21.15.12:0/390874417 learned_addr learned my addr 172.21.15.12:0/390874417 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca81feb640 1 --2- 172.21.15.12:0/390874417 >> v2:172.21.15.156:3300/0 conn(0x7fca7c099b90 0x7fca7c09a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.898+0000 7fca80fe9640 1 --2- 172.21.15.12:0/390874417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 0x7fca7c0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457356623 0 0) 0x7fca7c0cdf60 con 0x7fca7c15bb90 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003520 con 0x7fca7c15bb90 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335261572 0 0) 0x7fca64003520 con 0x7fca7c15bb90 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca80fe9640 1 -- 172.21.15.12:0/390874417 >> v2:172.21.15.156:3300/0 conn(0x7fca7c099b90 msgr2=0x7fca7c09a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca80fe9640 1 --2- 172.21.15.12:0/390874417 >> v2:172.21.15.156:3300/0 conn(0x7fca7c099b90 0x7fca7c09a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca80fe9640 1 -- 172.21.15.12:0/390874417 >> v1:172.21.15.12:6789/0 conn(0x7fca7c15bb90 legacy=0x7fca7c0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.899+0000 7fca80fe9640 1 -- 172.21.15.12:0/390874417 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0cc980 con 0x7fca7c0cc390 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca80fe9640 1 --2- 172.21.15.12:0/390874417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 0x7fca7c0cda20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca700097a0 tx=0x7fca7002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f50790a7b3420c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70037070 con 0x7fca7c0cc390 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca70030510 con 0x7fca7c0cc390 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca7bfff640 1 -- 172.21.15.12:0/390874417 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0019f0 con 0x7fca7c0cc390 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 -- 172.21.15.12:0/390874417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 msgr2=0x7fca7c0cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 --2- 172.21.15.12:0/390874417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 0x7fca7c0cda20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca700097a0 tx=0x7fca7002b790 comp rx=0 tx=0).stop 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 -- 172.21.15.12:0/390874417 shutdown_connections 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 --2- 172.21.15.12:0/390874417 >> v2:172.21.15.156:3300/0 conn(0x7fca7c099b90 0x7fca7c09a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 --2- 172.21.15.12:0/390874417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c0cc390 0x7fca7c0cda20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.900+0000 7fca83a4e640 1 -- 172.21.15.12:0/390874417 >> 172.21.15.12:0/390874417 conn(0x7fca7c0b3370 msgr2=0x7fca7c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.901+0000 7fca83a4e640 1 -- 172.21.15.12:0/390874417 shutdown_connections 2024-03-08T08:29:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.901+0000 7fca83a4e640 1 -- 172.21.15.12:0/390874417 wait complete. 2024-03-08T08:29:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 Processor -- start 2024-03-08T08:29:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 -- start start 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c099b90 0x7fca7c149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca7c0cc390 0x7fca7c14a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cd220 con 0x7fca7c15bb90 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0ccf40 con 0x7fca7c0cc390 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.902+0000 7fca83a4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c15bf70 con 0x7fca7c099b90 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49604/0 (socket says 172.21.15.12:49604) 2024-03-08T08:29:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 -- 172.21.15.12:0/2694335545 learned_addr learned my addr 172.21.15.12:0/2694335545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca817ea640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c099b90 0x7fca7c149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca80fe9640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca7c0cc390 0x7fca7c14a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 -- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c099b90 msgr2=0x7fca7c149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c099b90 0x7fca7c149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 -- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca7c0cc390 msgr2=0x7fca7c14a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca7c0cc390 0x7fca7c14a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 -- 172.21.15.12:0/2694335545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0b6120 con 0x7fca7c15bb90 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca81feb640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca7400e930 tx=0x7fca7400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74017070 con 0x7fca7c15bb90 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7400c980 con 0x7fca7c15bb90 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c15d940 con 0x7fca7c15bb90 2024-03-08T08:29:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.903+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400cd70 con 0x7fca7c15bb90 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.904+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c15df00 con 0x7fca7c15bb90 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.905+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c0749c0 con 0x7fca7c15bb90 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.905+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca7401a070 con 0x7fca7c15bb90 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.905+0000 7fca79ffb640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.905+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca7408a1d0 con 0x7fca7c15bb90 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.905+0000 7fca817ea640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.906+0000 7fca817ea640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 0x7fca6806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fca6c002230 tx=0x7fca6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:40.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:40.910+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fca74058a20 con 0x7fca7c15bb90 2024-03-08T08:29:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.089+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca7c0cdbc0 con 0x7fca7c15bb90 2024-03-08T08:29:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.089+0000 7fca79ffb640 1 -- 172.21.15.12:0/2694335545 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca7405cac0 con 0x7fca7c15bb90 2024-03-08T08:29:41.090 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.091+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 msgr2=0x7fca6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.091+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 0x7fca6806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fca6c002230 tx=0x7fca6c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.091+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 msgr2=0x7fca7c0b55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.091+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca7400e930 tx=0x7fca7400edf0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.091+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 shutdown_connections 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca7c15bb90 0x7fca7c0b55b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca7c0cc390 0x7fca7c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 --2- 172.21.15.12:0/2694335545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fca7c099b90 0x7fca7c149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 >> 172.21.15.12:0/2694335545 conn(0x7fca7c0b3370 msgr2=0x7fca7c070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 shutdown_connections 2024-03-08T08:29:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.092+0000 7fca83a4e640 1 -- 172.21.15.12:0/2694335545 wait complete. 2024-03-08T08:29:41.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.206+0000 7f9025b1e640 1 Processor -- start 2024-03-08T08:29:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.206+0000 7f9025b1e640 1 -- start start 2024-03-08T08:29:41.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.207+0000 7f9025b1e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 0x7f90200ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.207+0000 7f9025b1e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f90200cd430 0x7f90200b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.207+0000 7f9025b1e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90200b2ca0 con 0x7f902015aa30 2024-03-08T08:29:41.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.207+0000 7f9025b1e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902006de10 con 0x7f90200cd430 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.207+0000 7f9025b1e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90200b2f80 con 0x7f902015ae10 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901effd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 0x7f90200ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901f7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f902015aa30 0x7f902006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:52910/0 (socket says 172.21.15.12:52910) 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f90200cd430 0x7f90200b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901f7fe640 1 -- 172.21.15.12:0/1046191448 learned_addr learned my addr 172.21.15.12:0/1046191448 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901effd640 1 -- 172.21.15.12:0/1046191448 >> v2:172.21.15.156:3300/0 conn(0x7f90200cd430 msgr2=0x7f90200b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901effd640 1 --2- 172.21.15.12:0/1046191448 >> v2:172.21.15.156:3300/0 conn(0x7f90200cd430 0x7f90200b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901effd640 1 -- 172.21.15.12:0/1046191448 >> v1:172.21.15.156:6789/0 conn(0x7f902015aa30 legacy=0x7f902006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.208+0000 7f901effd640 1 -- 172.21.15.12:0/1046191448 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90200b3260 con 0x7f902015ae10 2024-03-08T08:29:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f901effd640 1 --2- 172.21.15.12:0/1046191448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 0x7f90200ccef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f901400ba40 tx=0x7f901402e080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab90a6be9d266bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f901e7fc640 1 -- 172.21.15.12:0/1046191448 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901403b070 con 0x7f902015ae10 2024-03-08T08:29:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f901e7fc640 1 -- 172.21.15.12:0/1046191448 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901402ee30 con 0x7f902015ae10 2024-03-08T08:29:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f901e7fc640 1 -- 172.21.15.12:0/1046191448 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014036640 con 0x7f902015ae10 2024-03-08T08:29:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 -- 172.21.15.12:0/1046191448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 msgr2=0x7f90200ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 --2- 172.21.15.12:0/1046191448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 0x7f90200ccef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f901400ba40 tx=0x7f901402e080 comp rx=0 tx=0).stop 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 -- 172.21.15.12:0/1046191448 shutdown_connections 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 --2- 172.21.15.12:0/1046191448 >> v2:172.21.15.156:3300/0 conn(0x7f90200cd430 0x7f90200b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 --2- 172.21.15.12:0/1046191448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f902015ae10 0x7f90200ccef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.209+0000 7f9025b1e640 1 -- 172.21.15.12:0/1046191448 >> 172.21.15.12:0/1046191448 conn(0x7f902006abd0 msgr2=0x7f902006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.210+0000 7f9025b1e640 1 -- 172.21.15.12:0/1046191448 shutdown_connections 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.210+0000 7f9025b1e640 1 -- 172.21.15.12:0/1046191448 wait complete. 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.211+0000 7f9025b1e640 1 Processor -- start 2024-03-08T08:29:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.211+0000 7f9025b1e640 1 -- start start 2024-03-08T08:29:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.211+0000 7f9025b1e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90200cd430 0x7f90201595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f902015aa30 0x7f902014ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902006b730 con 0x7f902015ae10 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90200b3820 con 0x7f902015aa30 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90200b4280 con 0x7f90200cd430 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49608/0 (socket says 172.21.15.12:49608) 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 -- 172.21.15.12:0/3361357031 learned_addr learned my addr 172.21.15.12:0/3361357031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901f7fe640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90200cd430 0x7f90201595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901effd640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f902015aa30 0x7f902014ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 -- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90200cd430 msgr2=0x7f90201595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90200cd430 0x7f90201595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 -- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f902015aa30 msgr2=0x7f902014ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f902015aa30 0x7f902014ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 -- 172.21.15.12:0/3361357031 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902014cf90 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f901ffff640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f901800efc0 tx=0x7f901800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9018018020 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9018014480 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902014f760 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9018014870 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.212+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9020156dc0 con 0x7f902015ae10 2024-03-08T08:29:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.213+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9020156be0 con 0x7f902015ae10 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.214+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9018005d10 con 0x7f902015ae10 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.214+0000 7f9007fff640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 0x7f900c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.214+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f901808ab80 con 0x7f902015ae10 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.214+0000 7f901f7fe640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 0x7f900c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.215+0000 7f901f7fe640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 0x7f900c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fd0 tx=0x7f9010007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.219+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9020156be0 con 0x7f902015ae10 2024-03-08T08:29:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.404+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90200749c0 con 0x7f902015ae10 2024-03-08T08:29:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.404+0000 7f9007fff640 1 -- 172.21.15.12:0/3361357031 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9018022070 con 0x7f902015ae10 2024-03-08T08:29:41.406 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 msgr2=0x7f900c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 0x7f900c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fd0 tx=0x7f9010007e80 comp rx=0 tx=0).stop 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 msgr2=0x7f902014f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f901800efc0 tx=0x7f901800c490 comp rx=0 tx=0).stop 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 shutdown_connections 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.406+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f900c06b4d0 0x7f900c06d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f902015ae10 0x7f902014f150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f902015aa30 0x7f902014ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 --2- 172.21.15.12:0/3361357031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90200cd430 0x7f90201595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 >> 172.21.15.12:0/3361357031 conn(0x7f902006abd0 msgr2=0x7f90200b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 shutdown_connections 2024-03-08T08:29:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.407+0000 7f9025b1e640 1 -- 172.21.15.12:0/3361357031 wait complete. 2024-03-08T08:29:41.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:41.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 Processor -- start 2024-03-08T08:29:41.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 -- start start 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feebc15dbc0 0x7feebc0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 0x7feebc0b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feebc0b7520 con 0x7feebc0b7080 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feebc0b7800 con 0x7feebc15dbc0 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.522+0000 7feec1607640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0b7ae0 con 0x7feebc0b9760 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feebc15dbc0 0x7feebc0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feebc15dbc0 0x7feebc0b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49610/0 (socket says 172.21.15.12:49610) 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 -- 172.21.15.12:0/476685318 learned_addr learned my addr 172.21.15.12:0/476685318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebb7fe640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 0x7feebc0b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 -- 172.21.15.12:0/476685318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 msgr2=0x7feebc0b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 0x7feebc0b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 -- 172.21.15.12:0/476685318 >> v1:172.21.15.156:6789/0 conn(0x7feebc0b7080 legacy=0x7feebc0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebaffd640 1 -- 172.21.15.12:0/476685318 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0b7dc0 con 0x7feebc15dbc0 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.523+0000 7feebb7fe640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 0x7feebc0b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feebaffd640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0b6b40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feeac009870 tx=0x7feeac02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd512ce6c373ff28 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feeba7fc640 1 -- 172.21.15.12:0/476685318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac02f840 con 0x7feebc15dbc0 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feeba7fc640 1 -- 172.21.15.12:0/476685318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac02f9e0 con 0x7feebc15dbc0 2024-03-08T08:29:41.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feeba7fc640 1 -- 172.21.15.12:0/476685318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac02fdb0 con 0x7feebc15dbc0 2024-03-08T08:29:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 -- 172.21.15.12:0/476685318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 msgr2=0x7feebc0b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0b6b40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feeac009870 tx=0x7feeac02ba20 comp rx=0 tx=0).stop 2024-03-08T08:29:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 -- 172.21.15.12:0/476685318 shutdown_connections 2024-03-08T08:29:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b9760 0x7feebc0b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 --2- 172.21.15.12:0/476685318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0b6b40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 -- 172.21.15.12:0/476685318 >> 172.21.15.12:0/476685318 conn(0x7feebc072720 msgr2=0x7feebc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 -- 172.21.15.12:0/476685318 shutdown_connections 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.524+0000 7feec1607640 1 -- 172.21.15.12:0/476685318 wait complete. 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 Processor -- start 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 -- start start 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b7080 0x7feebc0c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feebc0b9760 0x7feebc15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc15dfa0 con 0x7feebc15dbc0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0d2220 con 0x7feebc0b9760 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feec1607640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0b8b00 con 0x7feebc0b7080 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feebb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feebb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49622/0 (socket says 172.21.15.12:49622) 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.526+0000 7feebb7fe640 1 -- 172.21.15.12:0/2107483175 learned_addr learned my addr 172.21.15.12:0/2107483175 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebaffd640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b7080 0x7feebc0c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feeb3fff640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feebc0b9760 0x7feebc15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 -- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b7080 msgr2=0x7feebc0c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b7080 0x7feebc0c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 -- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feebc0b9760 msgr2=0x7feebc15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feebc0b9760 0x7feebc15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 -- 172.21.15.12:0/2107483175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0c71e0 con 0x7feebc15dbc0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feebb7fe640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009fa0 tx=0x7feeb400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb401a070 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb4003bb0 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc0cc700 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4004030 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.527+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc0cccc0 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.528+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc0749c0 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.528+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feeb401d070 con 0x7feebc15dbc0 2024-03-08T08:29:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.529+0000 7feeb37fe640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 0x7fee9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.529+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feeb408a490 con 0x7feebc15dbc0 2024-03-08T08:29:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.529+0000 7feebaffd640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 0x7fee9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.529+0000 7feebaffd640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 0x7fee9c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feeac036000 tx=0x7feeac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.533+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feebc0c71e0 con 0x7feebc15dbc0 2024-03-08T08:29:41.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.715+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feebc06b200 con 0x7feebc15dbc0 2024-03-08T08:29:41.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.716+0000 7feeb37fe640 1 -- 172.21.15.12:0/2107483175 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7feeb4010070 con 0x7feebc15dbc0 2024-03-08T08:29:41.717 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 msgr2=0x7fee9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 0x7fee9c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feeac036000 tx=0x7feeac008030 comp rx=0 tx=0).stop 2024-03-08T08:29:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 msgr2=0x7feebc0c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009fa0 tx=0x7feeb400d560 comp rx=0 tx=0).stop 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 shutdown_connections 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fee9c06b4d0 0x7fee9c06d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc15dbc0 0x7feebc0c8650 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feebc0b9760 0x7feebc15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 --2- 172.21.15.12:0/2107483175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feebc0b7080 0x7feebc0c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 >> 172.21.15.12:0/2107483175 conn(0x7feebc072720 msgr2=0x7feebc0ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 shutdown_connections 2024-03-08T08:29:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.718+0000 7feec1607640 1 -- 172.21.15.12:0/2107483175 wait complete. 2024-03-08T08:29:41.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:41.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.831+0000 7f1286365640 1 Processor -- start 2024-03-08T08:29:41.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.831+0000 7f1286365640 1 -- start start 2024-03-08T08:29:41.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f1286365640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f128015ddc0 0x7f12800d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f1286365640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 0x7f12800d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f1286365640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128015e1a0 con 0x7f12800d26f0 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f1286365640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12800b9030 con 0x7f128015ddc0 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f1286365640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800d13a0 con 0x7f12800d0f40 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f128015ddc0 0x7f12800d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f128015ddc0 0x7f12800d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49634/0 (socket says 172.21.15.12:49634) 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 -- 172.21.15.12:0/3302007875 learned_addr learned my addr 172.21.15.12:0/3302007875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127f7fe640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 0x7f12800d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 -- 172.21.15.12:0/3302007875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 msgr2=0x7f12800d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 0x7f12800d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 -- 172.21.15.12:0/3302007875 >> v1:172.21.15.156:6789/0 conn(0x7f12800d26f0 legacy=0x7f12800d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127ffff640 1 -- 172.21.15.12:0/3302007875 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800b8790 con 0x7f128015ddc0 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.832+0000 7f127f7fe640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 0x7f12800d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.833+0000 7f127ffff640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f128015ddc0 0x7f12800d0a00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1274009870 tx=0x7f127402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bba70e9cbc7192 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.833+0000 7f127effd640 1 -- 172.21.15.12:0/3302007875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274035020 con 0x7f128015ddc0 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.833+0000 7f127effd640 1 -- 172.21.15.12:0/3302007875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274031640 con 0x7f128015ddc0 2024-03-08T08:29:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.833+0000 7f127effd640 1 -- 172.21.15.12:0/3302007875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127402d040 con 0x7f128015ddc0 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 -- 172.21.15.12:0/3302007875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f128015ddc0 msgr2=0x7f12800d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f128015ddc0 0x7f12800d0a00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1274009870 tx=0x7f127402b8f0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 -- 172.21.15.12:0/3302007875 shutdown_connections 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f12800d0f40 0x7f12800d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 --2- 172.21.15.12:0/3302007875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f128015ddc0 0x7f12800d0a00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 -- 172.21.15.12:0/3302007875 >> 172.21.15.12:0/3302007875 conn(0x7f1280072720 msgr2=0x7f1280072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 -- 172.21.15.12:0/3302007875 shutdown_connections 2024-03-08T08:29:41.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.834+0000 7f1286365640 1 -- 172.21.15.12:0/3302007875 wait complete. 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.835+0000 7f1286365640 1 Processor -- start 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.835+0000 7f1286365640 1 -- start start 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12800d26f0 0x7f12800aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f128015ddc0 0x7f12800ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800b8d50 con 0x7f12800d0f40 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800d1d20 con 0x7f12800d26f0 2024-03-08T08:29:41.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800b9980 con 0x7f128015ddc0 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49650/0 (socket says 172.21.15.12:49650) 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 -- 172.21.15.12:0/3701958542 learned_addr learned my addr 172.21.15.12:0/3701958542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 -- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f128015ddc0 msgr2=0x7f12800ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f128015ddc0 0x7f12800ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 -- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12800d26f0 msgr2=0x7f12800aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12800d26f0 0x7f12800aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 -- 172.21.15.12:0/3701958542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128014dca0 con 0x7f12800d0f40 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127ffff640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1274000c00 tx=0x7f12740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274040070 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274039bf0 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.836+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128014df80 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.837+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274035040 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.837+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128014e540 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.837+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12800749c0 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.838+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f127404b060 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.838+0000 7f127cff9640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 0x7f126c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.838+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12740abce0 con 0x7f12800d0f40 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.838+0000 7f127f7fe640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 0x7f126c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:41.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.839+0000 7f127f7fe640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 0x7f126c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1270006fd0 tx=0x7f1270008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:41.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:41.843+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1274031610 con 0x7f12800d0f40 2024-03-08T08:29:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.035+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f12800d3f80 con 0x7f12800d0f40 2024-03-08T08:29:42.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.035+0000 7f127cff9640 1 -- 172.21.15.12:0/3701958542 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f127407ab60 con 0x7f12800d0f40 2024-03-08T08:29:42.037 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 msgr2=0x7f126c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 0x7f126c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1270006fd0 tx=0x7f1270008040 comp rx=0 tx=0).stop 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 msgr2=0x7f128014cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1274000c00 tx=0x7f12740046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 shutdown_connections 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f126c06b4d0 0x7f126c06d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f128015ddc0 0x7f12800ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12800d26f0 0x7f12800aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 --2- 172.21.15.12:0/3701958542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12800d0f40 0x7f128014cfe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.037+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 >> 172.21.15.12:0/3701958542 conn(0x7f1280072720 msgr2=0x7f12800b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.038+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 shutdown_connections 2024-03-08T08:29:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.038+0000 7f1286365640 1 -- 172.21.15.12:0/3701958542 wait complete. 2024-03-08T08:29:42.049 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-03-08T08:29:42.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:42.050 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-08T08:29:42.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:42.077 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:29:42.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:42.077 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-08T08:29:42.139 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:42.139 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:42.159 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:42.159 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:42.217 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 Processor -- start 2024-03-08T08:29:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 -- start start 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 0x7fc9740d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc9740d0f40 0x7fc9740d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97415e1a0 con 0x7fc9740d26f0 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9740b9030 con 0x7fc9740d0f40 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.318+0000 7fc97955c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740d13a0 con 0x7fc97415ddc0 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc96bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc9740d0f40 0x7fc9740d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc9737fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc9740d26f0 0x7fc9740d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44434/0 (socket says 172.21.15.12:44434) 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc9737fe640 1 -- 172.21.15.12:0/275471114 learned_addr learned my addr 172.21.15.12:0/275471114 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc972ffd640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 0x7fc9740d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc96bfff640 1 -- 172.21.15.12:0/275471114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 msgr2=0x7fc9740d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc96bfff640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 0x7fc9740d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc96bfff640 1 -- 172.21.15.12:0/275471114 >> v1:172.21.15.12:6789/0 conn(0x7fc9740d26f0 legacy=0x7fc9740d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc96bfff640 1 -- 172.21.15.12:0/275471114 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740b8790 con 0x7fc9740d0f40 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc9727fc640 1 -- 172.21.15.12:0/275471114 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950159962 0 0) 0x7fc97415e1a0 con 0x7fc9740d26f0 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.319+0000 7fc972ffd640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 0x7fc9740d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc96bfff640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d0f40 0x7fc9740d21b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0097a0 tx=0x7fc95c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dabf90a1428c9d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc9727fc640 1 -- 172.21.15.12:0/275471114 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c035020 con 0x7fc9740d0f40 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc9727fc640 1 -- 172.21.15.12:0/275471114 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c031830 con 0x7fc9740d0f40 2024-03-08T08:29:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc9727fc640 1 -- 172.21.15.12:0/275471114 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c031c00 con 0x7fc9740d0f40 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 -- 172.21.15.12:0/275471114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d0f40 msgr2=0x7fc9740d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d0f40 0x7fc9740d21b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0097a0 tx=0x7fc95c02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 -- 172.21.15.12:0/275471114 shutdown_connections 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d0f40 0x7fc9740d21b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 --2- 172.21.15.12:0/275471114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc97415ddc0 0x7fc9740d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 -- 172.21.15.12:0/275471114 >> 172.21.15.12:0/275471114 conn(0x7fc974072720 msgr2=0x7fc974072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 -- 172.21.15.12:0/275471114 shutdown_connections 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.320+0000 7fc97955c640 1 -- 172.21.15.12:0/275471114 wait complete. 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 Processor -- start 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 -- start start 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9740d0f40 0x7fc97415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc97415ddc0 0x7fc97415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740b8d50 con 0x7fc9740d26f0 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740d1d20 con 0x7fc97415ddc0 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc97955c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740b9980 con 0x7fc9740d0f40 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc96bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc96bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49668/0 (socket says 172.21.15.12:49668) 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.322+0000 7fc96bfff640 1 -- 172.21.15.12:0/2873728444 learned_addr learned my addr 172.21.15.12:0/2873728444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc972ffd640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9740d0f40 0x7fc97415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 -- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9740d0f40 msgr2=0x7fc97415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9740d0f40 0x7fc97415ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 -- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc97415ddc0 msgr2=0x7fc97415f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc97415ddc0 0x7fc97415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 -- 172.21.15.12:0/2873728444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740aa5e0 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96bfff640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009890 tx=0x7fc95c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c041070 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c03aa30 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c035040 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9740aa8c0 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.323+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9740aae80 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.324+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9740749c0 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.324+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc95c04c060 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.325+0000 7fc96b7fe640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 0x7fc95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.325+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc95c0abc00 con 0x7fc9740d26f0 2024-03-08T08:29:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.325+0000 7fc972ffd640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 0x7fc95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.325+0000 7fc972ffd640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 0x7fc95406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc96400b840 tx=0x7fc964008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.329+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc95c0b2080 con 0x7fc9740d26f0 2024-03-08T08:29:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.510+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9740d24d0 con 0x7fc9740d26f0 2024-03-08T08:29:42.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.511+0000 7fc96b7fe640 1 -- 172.21.15.12:0/2873728444 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fc95c07ab10 con 0x7fc9740d26f0 2024-03-08T08:29:42.512 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:42.512 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 msgr2=0x7fc95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 0x7fc95406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc96400b840 tx=0x7fc964008040 comp rx=0 tx=0).stop 2024-03-08T08:29:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 msgr2=0x7fc9740ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009890 tx=0x7fc95c0046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 shutdown_connections 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc95406b4d0 0x7fc95406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc97415ddc0 0x7fc97415f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9740d26f0 0x7fc9740ad0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 --2- 172.21.15.12:0/2873728444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9740d0f40 0x7fc97415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 >> 172.21.15.12:0/2873728444 conn(0x7fc974072720 msgr2=0x7fc97415d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 shutdown_connections 2024-03-08T08:29:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.513+0000 7fc97955c640 1 -- 172.21.15.12:0/2873728444 wait complete. 2024-03-08T08:29:42.525 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:42.525 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.628+0000 7fa014dac640 1 Processor -- start 2024-03-08T08:29:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.628+0000 7fa014dac640 1 -- start start 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa01015dec0 0x7fa010161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa010161ed0 0x7fa0100d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0100d0790 0x7fa0100b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100d2140 con 0x7fa010161ed0 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100b6be0 con 0x7fa0100d0790 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa014dac640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015e2a0 con 0x7fa01015dec0 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa010161ed0 0x7fa0100d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa010161ed0 0x7fa0100d0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49678/0 (socket says 172.21.15.12:49678) 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 -- 172.21.15.12:0/1754071228 learned_addr learned my addr 172.21.15.12:0/1754071228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00ed9d640 1 --2- 172.21.15.12:0/1754071228 >> v2:172.21.15.156:3300/0 conn(0x7fa0100d0790 0x7fa0100b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00e59c640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa01015dec0 0x7fa010161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 -- 172.21.15.12:0/1754071228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa01015dec0 msgr2=0x7fa010161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa01015dec0 0x7fa010161990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 -- 172.21.15.12:0/1754071228 >> v2:172.21.15.156:3300/0 conn(0x7fa0100d0790 msgr2=0x7fa0100b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/1754071228 >> v2:172.21.15.156:3300/0 conn(0x7fa0100d0790 0x7fa0100b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.629+0000 7fa00dd9b640 1 -- 172.21.15.12:0/1754071228 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100d0c00 con 0x7fa010161ed0 2024-03-08T08:29:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.630+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa010161ed0 0x7fa0100d0250 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa0100b3440 tx=0x7fa00402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32abab667e55b667 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.630+0000 7fa00d59a640 1 -- 172.21.15.12:0/1754071228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004035020 con 0x7fa010161ed0 2024-03-08T08:29:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.630+0000 7fa00d59a640 1 -- 172.21.15.12:0/1754071228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004031870 con 0x7fa010161ed0 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.630+0000 7fa00d59a640 1 -- 172.21.15.12:0/1754071228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004031c40 con 0x7fa010161ed0 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 -- 172.21.15.12:0/1754071228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa010161ed0 msgr2=0x7fa0100d0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa010161ed0 0x7fa0100d0250 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa0100b3440 tx=0x7fa00402b910 comp rx=0 tx=0).stop 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 -- 172.21.15.12:0/1754071228 shutdown_connections 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 --2- 172.21.15.12:0/1754071228 >> v2:172.21.15.156:3300/0 conn(0x7fa0100d0790 0x7fa0100b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa010161ed0 0x7fa0100d0250 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 --2- 172.21.15.12:0/1754071228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa01015dec0 0x7fa010161990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 -- 172.21.15.12:0/1754071228 >> 172.21.15.12:0/1754071228 conn(0x7fa01006c240 msgr2=0x7fa01006c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 -- 172.21.15.12:0/1754071228 shutdown_connections 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.631+0000 7fa014dac640 1 -- 172.21.15.12:0/1754071228 wait complete. 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.632+0000 7fa014dac640 1 Processor -- start 2024-03-08T08:29:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.632+0000 7fa014dac640 1 -- start start 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 0x7fa0101f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa010161ed0 0x7fa010160a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100d1dd0 con 0x7fa0100d0790 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa010162630 con 0x7fa01015dec0 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa014dac640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b94f0 con 0x7fa010161ed0 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49680/0 (socket says 172.21.15.12:49680) 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 -- 172.21.15.12:0/111449583 learned_addr learned my addr 172.21.15.12:0/111449583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 0x7fa0101f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00ed9d640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa010161ed0 0x7fa010160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 -- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa010161ed0 msgr2=0x7fa010160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa010161ed0 0x7fa010160a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 -- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 msgr2=0x7fa0101f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 0x7fa0101f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00e59c640 1 -- 172.21.15.12:0/111449583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01015f9a0 con 0x7fa0100d0790 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.633+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 0x7fa0101f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7fa00e59c640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa00800d830 tx=0x7fa00800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00800b6c0 con 0x7fa0100d0790 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa008019070 con 0x7fa0100d0790 2024-03-08T08:29:42.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01015fc80 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00800ba80 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.634+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa010160240 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.635+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0100749c0 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.635+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa00800bc20 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.636+0000 7f9ffeffd640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.636+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00808a650 con 0x7fa0100d0790 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.636+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.636+0000 7fa00dd9b640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa00403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.640+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa00801c360 con 0x7fa0100d0790 2024-03-08T08:29:42.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.820+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0100d1750 con 0x7fa0100d0790 2024-03-08T08:29:42.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.821+0000 7f9ffeffd640 1 -- 172.21.15.12:0/111449583 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa008059560 con 0x7fa0100d0790 2024-03-08T08:29:42.822 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:42.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 msgr2=0x7f9ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa00403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:42.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 msgr2=0x7fa0101f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa00800d830 tx=0x7fa00800dcf0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 shutdown_connections 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa010161ed0 0x7fa010160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa01015dec0 0x7fa0101f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 --2- 172.21.15.12:0/111449583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0100d0790 0x7fa0101f3d10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 >> 172.21.15.12:0/111449583 conn(0x7fa01006c240 msgr2=0x7fa0100b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 shutdown_connections 2024-03-08T08:29:42.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.823+0000 7fa014dac640 1 -- 172.21.15.12:0/111449583 wait complete. 2024-03-08T08:29:42.835 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.938+0000 7fcf8a447640 1 Processor -- start 2024-03-08T08:29:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.938+0000 7fcf8a447640 1 -- start start 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf8a447640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcf8415e0d0 0x7fcf840cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf8a447640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf8406bef0 0x7fcf840cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf8a447640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf840d0290 con 0x7fcf84099b90 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf8a447640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840d0570 con 0x7fcf8406bef0 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf8a447640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840d0850 con 0x7fcf8415e0d0 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf8406bef0 0x7fcf840cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf8406bef0 0x7fcf840cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49688/0 (socket says 172.21.15.12:49688) 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 -- 172.21.15.12:0/423458869 learned_addr learned my addr 172.21.15.12:0/423458869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf837fe640 1 --2- 172.21.15.12:0/423458869 >> v2:172.21.15.156:3300/0 conn(0x7fcf8415e0d0 0x7fcf840cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 -- 172.21.15.12:0/423458869 >> v2:172.21.15.156:3300/0 conn(0x7fcf8415e0d0 msgr2=0x7fcf840cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 --2- 172.21.15.12:0/423458869 >> v2:172.21.15.156:3300/0 conn(0x7fcf8415e0d0 0x7fcf840cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 -- 172.21.15.12:0/423458869 >> v1:172.21.15.156:6789/0 conn(0x7fcf84099b90 legacy=0x7fcf84099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:42.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.939+0000 7fcf889e4640 1 -- 172.21.15.12:0/423458869 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840d0b30 con 0x7fcf8406bef0 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.940+0000 7fcf889e4640 1 --2- 172.21.15.12:0/423458869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8406bef0 0x7fcf840cbfc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf840c89b0 tx=0x7fcf7c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3120be0174b266a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.940+0000 7fcf82ffd640 1 -- 172.21.15.12:0/423458869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c02c9f0 con 0x7fcf8406bef0 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.940+0000 7fcf82ffd640 1 -- 172.21.15.12:0/423458869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c02cb90 con 0x7fcf8406bef0 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.940+0000 7fcf82ffd640 1 -- 172.21.15.12:0/423458869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0343f0 con 0x7fcf8406bef0 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 -- 172.21.15.12:0/423458869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8406bef0 msgr2=0x7fcf840cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 --2- 172.21.15.12:0/423458869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8406bef0 0x7fcf840cbfc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf840c89b0 tx=0x7fcf7c02be50 comp rx=0 tx=0).stop 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 -- 172.21.15.12:0/423458869 shutdown_connections 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 --2- 172.21.15.12:0/423458869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf8406bef0 0x7fcf840cbfc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 --2- 172.21.15.12:0/423458869 >> v2:172.21.15.156:3300/0 conn(0x7fcf8415e0d0 0x7fcf840cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 -- 172.21.15.12:0/423458869 >> 172.21.15.12:0/423458869 conn(0x7fcf840b3460 msgr2=0x7fcf840b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 -- 172.21.15.12:0/423458869 shutdown_connections 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.941+0000 7fcf8a447640 1 -- 172.21.15.12:0/423458869 wait complete. 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.942+0000 7fcf8a447640 1 Processor -- start 2024-03-08T08:29:42.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.942+0000 7fcf8a447640 1 -- start start 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 0x7fcf8415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 0x7fcf84158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8409b310 con 0x7fcf84152ca0 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8409b190 con 0x7fcf84099b90 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf8a447640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf840d1330 con 0x7fcf8406bef0 2024-03-08T08:29:42.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49692/0 (socket says 172.21.15.12:49692) 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 -- 172.21.15.12:0/1221091109 learned_addr learned my addr 172.21.15.12:0/1221091109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf837fe640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 0x7fcf84158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf83fff640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 0x7fcf8415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 -- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 msgr2=0x7fcf8415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 0x7fcf8415c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 -- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 msgr2=0x7fcf84158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 0x7fcf84158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 -- 172.21.15.12:0/1221091109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8415ce60 con 0x7fcf84152ca0 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf837fe640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 0x7fcf84158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf83fff640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 0x7fcf8415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.943+0000 7fcf889e4640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c036040 tx=0x7fcf7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.944+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c043020 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.944+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c03e450 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.944+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf84160710 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.944+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c03e860 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.944+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf84160cd0 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.945+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf840749c0 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.945+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf7c046020 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.945+0000 7fcf80ff9640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 0x7fcf6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.945+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf7c0acae0 con 0x7fcf84152ca0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.945+0000 7fcf83fff640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 0x7fcf6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.946+0000 7fcf83fff640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 0x7fcf6406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c003eb0 tx=0x7fcf6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:42.950+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcf7c04d030 con 0x7fcf84152ca0 2024-03-08T08:29:43.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.127+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf840b78f0 con 0x7fcf84152ca0 2024-03-08T08:29:43.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.127+0000 7fcf80ff9640 1 -- 172.21.15.12:0/1221091109 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf7c07b9f0 con 0x7fcf84152ca0 2024-03-08T08:29:43.128 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:43.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 msgr2=0x7fcf6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 0x7fcf6406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c003eb0 tx=0x7fcf6c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 msgr2=0x7fcf840b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c036040 tx=0x7fcf7c004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 shutdown_connections 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcf6406b4d0 0x7fcf6406d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf84152ca0 0x7fcf840b2320 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf84099b90 0x7fcf84158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 --2- 172.21.15.12:0/1221091109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcf8406bef0 0x7fcf8415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 >> 172.21.15.12:0/1221091109 conn(0x7fcf840b3460 msgr2=0x7fcf840cd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 shutdown_connections 2024-03-08T08:29:43.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.129+0000 7fcf8a447640 1 -- 172.21.15.12:0/1221091109 wait complete. 2024-03-08T08:29:43.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.244+0000 7fa35b82e640 1 Processor -- start 2024-03-08T08:29:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.244+0000 7fa35b82e640 1 -- start start 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa35b82e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa3540b85c0 0x7fa3540b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa35b82e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540b8f30 con 0x7fa3540b9760 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa35b82e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540b7250 con 0x7fa35415dbc0 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa35b82e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3540b7520 con 0x7fa3540b85c0 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa358dc9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa3540b9760 0x7fa3540d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44436/0 (socket says 172.21.15.12:44436) 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa358dc9640 1 -- 172.21.15.12:0/12374770 learned_addr learned my addr 172.21.15.12:0/12374770 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852767395 0 0) 0x7fa3540b8f30 con 0x7fa3540b9760 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340003520 con 0x7fa3540b9760 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa359dcb640 1 --2- 172.21.15.12:0/12374770 >> v2:172.21.15.156:3300/0 conn(0x7fa3540b85c0 0x7fa3540b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231626419 0 0) 0x7fa340003520 con 0x7fa3540b9760 2024-03-08T08:29:43.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 >> v2:172.21.15.156:3300/0 conn(0x7fa3540b85c0 msgr2=0x7fa3540b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 --2- 172.21.15.12:0/12374770 >> v2:172.21.15.156:3300/0 conn(0x7fa3540b85c0 0x7fa3540b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 >> v1:172.21.15.156:6789/0 conn(0x7fa35415dbc0 legacy=0x7fa3540b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540b7800 con 0x7fa3540b9760 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fa344001670 con 0x7fa3540b9760 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 >> v1:172.21.15.12:6789/0 conn(0x7fa3540b9760 legacy=0x7fa3540d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.245+0000 7fa34bfff640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3400039b0 0x7fa340003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa34bfff640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa340004800 0x7fa340004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa34bfff640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa34000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b7520 con 0x7fa3400051c0 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34000c070 con 0x7fa3400039b0 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34000c270 con 0x7fa340004800 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa34000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa3595ca640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3400039b0 0x7fa340003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa358dc9640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa340004800 0x7fa340004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 -- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa340004800 msgr2=0x7fa340004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa340004800 0x7fa340004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 -- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3400039b0 msgr2=0x7fa340003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3400039b0 0x7fa340003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.246+0000 7fa359dcb640 1 -- 172.21.15.12:0/12374770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340003520 con 0x7fa3400051c0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa359dcb640 1 -- 172.21.15.12:0/12374770 reap_dead start 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa359dcb640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa34000b9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009fc0 tx=0x7fa340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b04368f77395d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c009c10 con 0x7fa3400051c0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c02cca0 con 0x7fa3400051c0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa34bfff640 1 -- 172.21.15.12:0/12374770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c003bd0 con 0x7fa3400051c0 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa35b82e640 1 -- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 msgr2=0x7fa34000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa35b82e640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa34000b9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009fc0 tx=0x7fa340005830 comp rx=0 tx=0).stop 2024-03-08T08:29:43.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa35b82e640 1 -- 172.21.15.12:0/12374770 shutdown_connections 2024-03-08T08:29:43.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa35b82e640 1 --2- 172.21.15.12:0/12374770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa34000b9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.247+0000 7fa35b82e640 1 -- 172.21.15.12:0/12374770 >> 172.21.15.12:0/12374770 conn(0x7fa354072720 msgr2=0x7fa354072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.248+0000 7fa35b82e640 1 -- 172.21.15.12:0/12374770 shutdown_connections 2024-03-08T08:29:43.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.248+0000 7fa35b82e640 1 -- 172.21.15.12:0/12374770 wait complete. 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 Processor -- start 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 -- start start 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3541f4e00 0x7fa3540ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3540acda0 0x7fa3540ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b80a0 con 0x7fa3400051c0 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b7dc0 con 0x7fa3540acda0 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.249+0000 7fa35b82e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35415dfa0 con 0x7fa3541f4e00 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49700/0 (socket says 172.21.15.12:49700) 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 -- 172.21.15.12:0/1061855337 learned_addr learned my addr 172.21.15.12:0/1061855337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 -- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3541f4e00 msgr2=0x7fa3540ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa358dc9640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3541f4e00 0x7fa3540ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa359dcb640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3540acda0 0x7fa3540ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3541f4e00 0x7fa3540ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 -- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3540acda0 msgr2=0x7fa3540ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3540acda0 0x7fa3540ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 -- 172.21.15.12:0/1061855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540cbe00 con 0x7fa3400051c0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa3595ca640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa35000f5f0 tx=0x7fa35000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3500052e0 con 0x7fa3400051c0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350018070 con 0x7fa3400051c0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3540cc0e0 con 0x7fa3400051c0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350003810 con 0x7fa3400051c0 2024-03-08T08:29:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.250+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3540cd100 con 0x7fa3400051c0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.251+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa354068500 con 0x7fa3400051c0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.251+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3500039b0 con 0x7fa3400051c0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.252+0000 7fa349ffb640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.252+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa350089f80 con 0x7fa3400051c0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.252+0000 7fa358dc9640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.252+0000 7fa358dc9640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 0x7fa33806d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa3541f6c50 tx=0x7fa344009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.256+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa354068500 con 0x7fa3400051c0 2024-03-08T08:29:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.437+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3540ad1b0 con 0x7fa3400051c0 2024-03-08T08:29:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.437+0000 7fa349ffb640 1 -- 172.21.15.12:0/1061855337 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3540ad1b0 con 0x7fa3400051c0 2024-03-08T08:29:43.438 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 msgr2=0x7fa33806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 0x7fa33806d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa3541f6c50 tx=0x7fa344009040 comp rx=0 tx=0).stop 2024-03-08T08:29:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 msgr2=0x7fa3541f48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa35000f5f0 tx=0x7fa35000fab0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 shutdown_connections 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3540acda0 0x7fa3540ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3541f4e00 0x7fa3540ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 --2- 172.21.15.12:0/1061855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3400051c0 0x7fa3541f48c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 >> 172.21.15.12:0/1061855337 conn(0x7fa354072720 msgr2=0x7fa3540d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 shutdown_connections 2024-03-08T08:29:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.439+0000 7fa35b82e640 1 -- 172.21.15.12:0/1061855337 wait complete. 2024-03-08T08:29:43.451 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.552+0000 7feb32b63640 1 Processor -- start 2024-03-08T08:29:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.552+0000 7feb32b63640 1 -- start start 2024-03-08T08:29:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb32b63640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb2c0d1110 0x7feb2c0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb32b63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb32b63640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c15e1a0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb32b63640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0b9030 con 0x7feb2c0d1110 2024-03-08T08:29:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb32b63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d1d20 con 0x7feb2c0d31c0 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb2c0d1110 0x7feb2c0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb308ff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feb2c15ddc0 0x7feb2c0d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44438/0 (socket says 172.21.15.12:44438) 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb308ff640 1 -- 172.21.15.12:0/1539539339 learned_addr learned my addr 172.21.15.12:0/1539539339 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2bfff640 1 -- 172.21.15.12:0/1539539339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 msgr2=0x7feb2c0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2bfff640 1 --2- 172.21.15.12:0/1539539339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2b7fe640 1 -- 172.21.15.12:0/1539539339 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844032132 0 0) 0x7feb2c15e1a0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2bfff640 1 -- 172.21.15.12:0/1539539339 >> v1:172.21.15.12:6789/0 conn(0x7feb2c15ddc0 legacy=0x7feb2c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.553+0000 7feb2bfff640 1 -- 172.21.15.12:0/1539539339 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0b8790 con 0x7feb2c0d1110 2024-03-08T08:29:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.554+0000 7feb2bfff640 1 --2- 172.21.15.12:0/1539539339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c0d1110 0x7feb2c0b6be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feb180097a0 tx=0x7feb1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e448d56d424834ce server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.554+0000 7feb2b7fe640 1 -- 172.21.15.12:0/1539539339 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18035020 con 0x7feb2c0d1110 2024-03-08T08:29:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.554+0000 7feb2b7fe640 1 -- 172.21.15.12:0/1539539339 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18031830 con 0x7feb2c0d1110 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.554+0000 7feb2b7fe640 1 -- 172.21.15.12:0/1539539339 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18031c00 con 0x7feb2c0d1110 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 -- 172.21.15.12:0/1539539339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c0d1110 msgr2=0x7feb2c0b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 --2- 172.21.15.12:0/1539539339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c0d1110 0x7feb2c0b6be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feb180097a0 tx=0x7feb1802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 -- 172.21.15.12:0/1539539339 shutdown_connections 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 --2- 172.21.15.12:0/1539539339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c0b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 --2- 172.21.15.12:0/1539539339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c0d1110 0x7feb2c0b6be0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 -- 172.21.15.12:0/1539539339 >> 172.21.15.12:0/1539539339 conn(0x7feb2c072720 msgr2=0x7feb2c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 -- 172.21.15.12:0/1539539339 shutdown_connections 2024-03-08T08:29:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.555+0000 7feb32b63640 1 -- 172.21.15.12:0/1539539339 wait complete. 2024-03-08T08:29:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.556+0000 7feb32b63640 1 Processor -- start 2024-03-08T08:29:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.556+0000 7feb32b63640 1 -- start start 2024-03-08T08:29:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2c0d1110 0x7feb2c1e8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c1e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b9980 con 0x7feb2c15ddc0 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b8d50 con 0x7feb2c0d1110 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b7660 con 0x7feb2c0d31c0 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49718/0 (socket says 172.21.15.12:49718) 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 -- 172.21.15.12:0/4220589080 learned_addr learned my addr 172.21.15.12:0/4220589080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb308ff640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2c0d1110 0x7feb2c1e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb2bfff640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c1e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 -- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 msgr2=0x7feb2c1e73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c1e73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 -- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2c0d1110 msgr2=0x7feb2c1e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2c0d1110 0x7feb2c1e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 -- 172.21.15.12:0/4220589080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c1e7910 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb31100640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b9a0 tx=0x7feb2400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24019070 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2400cbc0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24014440 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c1ea2d0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.557+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c1e7e90 con 0x7feb2c15ddc0 2024-03-08T08:29:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.558+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c0749c0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.559+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2401c070 con 0x7feb2c15ddc0 2024-03-08T08:29:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.559+0000 7feb297fa640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 0x7feb1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.559+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb24089ef0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.559+0000 7feb308ff640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 0x7feb1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.560+0000 7feb308ff640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 0x7feb1006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f60 tx=0x7feb20007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.564+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb24058e00 con 0x7feb2c15ddc0 2024-03-08T08:29:43.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.757+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feb2c1615e0 con 0x7feb2c15ddc0 2024-03-08T08:29:43.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.757+0000 7feb297fa640 1 -- 172.21.15.12:0/4220589080 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7feb24058700 con 0x7feb2c15ddc0 2024-03-08T08:29:43.758 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 msgr2=0x7feb1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 0x7feb1006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f60 tx=0x7feb20007c00 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 msgr2=0x7feb2c160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b9a0 tx=0x7feb2400be60 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 shutdown_connections 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7feb1006b4d0 0x7feb1006d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb2c15ddc0 0x7feb2c160560 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2c0d31c0 0x7feb2c1e73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 --2- 172.21.15.12:0/4220589080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2c0d1110 0x7feb2c1e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 >> 172.21.15.12:0/4220589080 conn(0x7feb2c072720 msgr2=0x7feb2c0d3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.759+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 shutdown_connections 2024-03-08T08:29:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.760+0000 7feb32b63640 1 -- 172.21.15.12:0/4220589080 wait complete. 2024-03-08T08:29:43.771 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:43.771 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:43.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.891+0000 7f7514415640 1 Processor -- start 2024-03-08T08:29:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.891+0000 7f7514415640 1 -- start start 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f7514415640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f750c15ddc0 0x7f750c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f7514415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c0b7bf0 0x7f750c0b8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f7514415640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c15e1a0 con 0x7f750c0d0f40 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f7514415640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0b9030 con 0x7f750c15ddc0 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f7514415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0d1d20 con 0x7f750c0b7bf0 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75121b1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f750c15ddc0 0x7f750c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75119b0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f750c0d0f40 0x7f750c0d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44440/0 (socket says 172.21.15.12:44440) 2024-03-08T08:29:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75119b0640 1 -- 172.21.15.12:0/3037642631 learned_addr learned my addr 172.21.15.12:0/3037642631 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75129b2640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c0b7bf0 0x7f750c0b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75121b1640 1 -- 172.21.15.12:0/3037642631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c0b7bf0 msgr2=0x7f750c0b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75121b1640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c0b7bf0 0x7f750c0b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75121b1640 1 -- 172.21.15.12:0/3037642631 >> v1:172.21.15.12:6789/0 conn(0x7f750c0d0f40 legacy=0x7f750c0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75121b1640 1 -- 172.21.15.12:0/3037642631 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0b8790 con 0x7f750c15ddc0 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.892+0000 7f75111af640 1 -- 172.21.15.12:0/3037642631 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299272335 0 0) 0x7f750c15e1a0 con 0x7f750c0d0f40 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.893+0000 7f75121b1640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c15ddc0 0x7f750c0d0a00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f75080097a0 tx=0x7f750802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580f31d9bae857f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.893+0000 7f75111af640 1 -- 172.21.15.12:0/3037642631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508035020 con 0x7f750c15ddc0 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.893+0000 7f75111af640 1 -- 172.21.15.12:0/3037642631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508031830 con 0x7f750c15ddc0 2024-03-08T08:29:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.893+0000 7f75111af640 1 -- 172.21.15.12:0/3037642631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508031c00 con 0x7f750c15ddc0 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 -- 172.21.15.12:0/3037642631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c15ddc0 msgr2=0x7f750c0d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c15ddc0 0x7f750c0d0a00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f75080097a0 tx=0x7f750802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 -- 172.21.15.12:0/3037642631 shutdown_connections 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c0b7bf0 0x7f750c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 --2- 172.21.15.12:0/3037642631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c15ddc0 0x7f750c0d0a00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 -- 172.21.15.12:0/3037642631 >> 172.21.15.12:0/3037642631 conn(0x7f750c072720 msgr2=0x7f750c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:43.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 -- 172.21.15.12:0/3037642631 shutdown_connections 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.894+0000 7f7514415640 1 -- 172.21.15.12:0/3037642631 wait complete. 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.895+0000 7f7514415640 1 Processor -- start 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.895+0000 7f7514415640 1 -- start start 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.895+0000 7f7514415640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.895+0000 7f7514415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f750c0d0f40 0x7f750c15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.895+0000 7f7514415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c15ddc0 0x7f750c0ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7514415640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b9980 con 0x7f750c0b7bf0 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7514415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b8d50 con 0x7f750c0d0f40 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7514415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0d28c0 con 0x7f750c15ddc0 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49730/0 (socket says 172.21.15.12:49730) 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 -- 172.21.15.12:0/3004142328 learned_addr learned my addr 172.21.15.12:0/3004142328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:43.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75119b0640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f750c0d0f40 0x7f750c15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75129b2640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c15ddc0 0x7f750c0ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 -- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c15ddc0 msgr2=0x7f750c0ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c15ddc0 0x7f750c0ab570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 -- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f750c0d0f40 msgr2=0x7f750c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f750c0d0f40 0x7f750c15fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 -- 172.21.15.12:0/3004142328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0abf80 con 0x7f750c0b7bf0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f75121b1640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f75080098c0 tx=0x7f75080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508040070 con 0x7f750c0b7bf0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750803e070 con 0x7f750c0b7bf0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508035040 con 0x7f750c0b7bf0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c0ac260 con 0x7f750c0b7bf0 2024-03-08T08:29:43.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.896+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c0ac820 con 0x7f750c0b7bf0 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.897+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c0749c0 con 0x7f750c0b7bf0 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.898+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7508038020 con 0x7f750c0b7bf0 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.898+0000 7f7502ffd640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 0x7f74f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.898+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75080ab810 con 0x7f750c0b7bf0 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.898+0000 7f75119b0640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 0x7f74f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.898+0000 7f75119b0640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 0x7f74f006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f750c161700 tx=0x7f74fc007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:43.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:43.902+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f750807a6a0 con 0x7f750c0b7bf0 2024-03-08T08:29:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.083+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f750c06cdb0 con 0x7f750c0b7bf0 2024-03-08T08:29:44.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.084+0000 7f7502ffd640 1 -- 172.21.15.12:0/3004142328 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f7508079fa0 con 0x7f750c0b7bf0 2024-03-08T08:29:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:44.086 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:44.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 msgr2=0x7f74f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 0x7f74f006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f750c161700 tx=0x7f74fc007b50 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 msgr2=0x7f750c0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f75080098c0 tx=0x7f75080046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 shutdown_connections 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f74f006b4d0 0x7f74f006d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f750c15ddc0 0x7f750c0ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f750c0d0f40 0x7f750c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 --2- 172.21.15.12:0/3004142328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f750c0b7bf0 0x7f750c0aff80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 >> 172.21.15.12:0/3004142328 conn(0x7f750c072720 msgr2=0x7f750c0d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 shutdown_connections 2024-03-08T08:29:44.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.086+0000 7f7514415640 1 -- 172.21.15.12:0/3004142328 wait complete. 2024-03-08T08:29:44.098 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:44.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:44.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.201+0000 7fe64f781640 1 Processor -- start 2024-03-08T08:29:44.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.201+0000 7fe64f781640 1 -- start start 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.201+0000 7fe64f781640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe64815dbc0 0x7fe6480b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64f781640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe6480b7080 0x7fe6480d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64f781640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6480d2050 0x7fe6480b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64f781640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480b7520 con 0x7fe6480d2050 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64f781640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480b7800 con 0x7fe6480b7080 2024-03-08T08:29:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64f781640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480b7ae0 con 0x7fe64815dbc0 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6480d2050 0x7fe6480b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6480d2050 0x7fe6480b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49744/0 (socket says 172.21.15.12:49744) 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 -- 172.21.15.12:0/48841481 learned_addr learned my addr 172.21.15.12:0/48841481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64d51d640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe64815dbc0 0x7fe6480b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64cd1c640 1 --2- 172.21.15.12:0/48841481 >> v2:172.21.15.156:3300/0 conn(0x7fe6480b7080 0x7fe6480d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 -- 172.21.15.12:0/48841481 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe64815dbc0 msgr2=0x7fe6480b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe64815dbc0 0x7fe6480b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 -- 172.21.15.12:0/48841481 >> v2:172.21.15.156:3300/0 conn(0x7fe6480b7080 msgr2=0x7fe6480d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 --2- 172.21.15.12:0/48841481 >> v2:172.21.15.156:3300/0 conn(0x7fe6480b7080 0x7fe6480d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.202+0000 7fe64dd1e640 1 -- 172.21.15.12:0/48841481 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480b7dc0 con 0x7fe6480d2050 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64dd1e640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480b85c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d4ae0 tx=0x7fe64002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321c2d2c67438b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe63ffff640 1 -- 172.21.15.12:0/48841481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640034070 con 0x7fe6480d2050 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe63ffff640 1 -- 172.21.15.12:0/48841481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe640030860 con 0x7fe6480d2050 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe63ffff640 1 -- 172.21.15.12:0/48841481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640030c30 con 0x7fe6480d2050 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 -- 172.21.15.12:0/48841481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 msgr2=0x7fe6480b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480b85c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d4ae0 tx=0x7fe64002b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 -- 172.21.15.12:0/48841481 shutdown_connections 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480b85c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 --2- 172.21.15.12:0/48841481 >> v2:172.21.15.156:3300/0 conn(0x7fe6480b7080 0x7fe6480d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 --2- 172.21.15.12:0/48841481 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe64815dbc0 0x7fe6480b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.203+0000 7fe64f781640 1 -- 172.21.15.12:0/48841481 >> 172.21.15.12:0/48841481 conn(0x7fe648072720 msgr2=0x7fe648072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.204+0000 7fe64f781640 1 -- 172.21.15.12:0/48841481 shutdown_connections 2024-03-08T08:29:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.204+0000 7fe64f781640 1 -- 172.21.15.12:0/48841481 wait complete. 2024-03-08T08:29:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 Processor -- start 2024-03-08T08:29:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 -- start start 2024-03-08T08:29:44.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6480b7080 0x7fe6481f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe64815dbc0 0x7fe6480c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480b9760 con 0x7fe6480d2050 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64815dfa0 con 0x7fe64815dbc0 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64f781640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480b98e0 con 0x7fe6480b7080 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64cd1c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64cd1c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49746/0 (socket says 172.21.15.12:49746) 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.205+0000 7fe64cd1c640 1 -- 172.21.15.12:0/353515335 learned_addr learned my addr 172.21.15.12:0/353515335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64dd1e640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe64815dbc0 0x7fe6480c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64d51d640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6480b7080 0x7fe6481f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 -- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6480b7080 msgr2=0x7fe6481f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6480b7080 0x7fe6481f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 -- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe64815dbc0 msgr2=0x7fe6480c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe64815dbc0 0x7fe6480c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 -- 172.21.15.12:0/353515335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480ab020 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64cd1c640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe63800c7f0 tx=0x7fe63800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638015070 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe638004220 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6481f46b0 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638007620 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.206+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6480aac60 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.207+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6480749c0 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.208+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6380079d0 con 0x7fe6480d2050 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.208+0000 7fe63dffb640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 0x7fe62c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.208+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe63808a0b0 con 0x7fe6480d2050 2024-03-08T08:29:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.208+0000 7fe64d51d640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 0x7fe62c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.209+0000 7fe64d51d640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 0x7fe62c06d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe6480c9790 tx=0x7fe6440077b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.212+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe638058970 con 0x7fe6480d2050 2024-03-08T08:29:44.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.391+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe64806b200 con 0x7fe6480d2050 2024-03-08T08:29:44.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.392+0000 7fe63dffb640 1 -- 172.21.15.12:0/353515335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe63805ca10 con 0x7fe6480d2050 2024-03-08T08:29:44.393 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 msgr2=0x7fe62c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 0x7fe62c06d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe6480c9790 tx=0x7fe6440077b0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 msgr2=0x7fe6480aa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe63800c7f0 tx=0x7fe63800ccb0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 shutdown_connections 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe62c06b500 0x7fe62c06d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe64815dbc0 0x7fe6480c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6480d2050 0x7fe6480aa720 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 --2- 172.21.15.12:0/353515335 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6480b7080 0x7fe6481f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 >> 172.21.15.12:0/353515335 conn(0x7fe648072720 msgr2=0x7fe6480c98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 shutdown_connections 2024-03-08T08:29:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.394+0000 7fe64f781640 1 -- 172.21.15.12:0/353515335 wait complete. 2024-03-08T08:29:44.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:44.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:29:44.451 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:29:44.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.509+0000 7f9f67c95640 1 Processor -- start 2024-03-08T08:29:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.510+0000 7f9f67c95640 1 -- start start 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.510+0000 7f9f67c95640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9f6006bef0 0x7f9f600cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.510+0000 7f9f67c95640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600d0290 con 0x7f9f60099b90 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.510+0000 7f9f67c95640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600d0570 con 0x7f9f6015e0d0 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.510+0000 7f9f67c95640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600d0850 con 0x7f9f6006bef0 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f65a31640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9f60099b90 0x7f9f60099f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44444/0 (socket says 172.21.15.12:44444) 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f65a31640 1 -- 172.21.15.12:0/4203159380 learned_addr learned my addr 172.21.15.12:0/4203159380 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f66232640 1 --2- 172.21.15.12:0/4203159380 >> v2:172.21.15.156:3300/0 conn(0x7f9f6006bef0 0x7f9f600cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705766671 0 0) 0x7f9f600d0290 con 0x7f9f60099b90 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c003520 con 0x7f9f60099b90 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1872593421 0 0) 0x7f9f4c003520 con 0x7f9f60099b90 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 >> v2:172.21.15.156:3300/0 conn(0x7f9f6006bef0 msgr2=0x7f9f600cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 --2- 172.21.15.12:0/4203159380 >> v2:172.21.15.156:3300/0 conn(0x7f9f6006bef0 0x7f9f600cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 >> v1:172.21.15.156:6789/0 conn(0x7f9f6015e0d0 legacy=0x7f9f600cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600d0b30 con 0x7f9f60099b90 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f9f54001670 con 0x7f9f60099b90 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f66232640 1 --2- 172.21.15.12:0/4203159380 >> v2:172.21.15.156:3300/0 conn(0x7f9f6006bef0 0x7f9f600cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 >> v1:172.21.15.12:6789/0 conn(0x7f9f60099b90 legacy=0x7f9f60099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f4c0039b0 0x7f9f4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 0x7f9f4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c0051c0 0x7f9f4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600d0850 con 0x7f9f4c004800 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c00c060 con 0x7f9f4c0039b0 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c00c260 con 0x7f9f4c0051c0 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f65230640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 0x7f9f4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.511+0000 7f9f65a31640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f4c0039b0 0x7f9f4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f66232640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c0051c0 0x7f9f4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 -- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c0051c0 msgr2=0x7f9f4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c0051c0 0x7f9f4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 -- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f4c0039b0 msgr2=0x7f9f4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f4c0039b0 0x7f9f4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 -- 172.21.15.12:0/4203159380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c003520 con 0x7f9f4c004800 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f66232640 1 -- 172.21.15.12:0/4203159380 reap_dead start 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.512+0000 7f9f65230640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 0x7f9f4c004c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c001750 tx=0x7f9f4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c06dead9dfbd43 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c001370 con 0x7f9f4c004800 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c0052d0 con 0x7f9f4c004800 2024-03-08T08:29:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f64a2f640 1 -- 172.21.15.12:0/4203159380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c003830 con 0x7f9f4c004800 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 -- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 msgr2=0x7f9f4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 0x7f9f4c004c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c001750 tx=0x7f9f4c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 -- 172.21.15.12:0/4203159380 shutdown_connections 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 --2- 172.21.15.12:0/4203159380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f4c004800 0x7f9f4c004c80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 -- 172.21.15.12:0/4203159380 >> 172.21.15.12:0/4203159380 conn(0x7f9f600b3460 msgr2=0x7f9f600b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 -- 172.21.15.12:0/4203159380 shutdown_connections 2024-03-08T08:29:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.513+0000 7f9f67c95640 1 -- 172.21.15.12:0/4203159380 wait complete. 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.514+0000 7f9f67c95640 1 Processor -- start 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.514+0000 7f9f67c95640 1 -- start start 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 0x7f9f600c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 0x7f9f600b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6009b310 con 0x7f9f601587c0 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6009b190 con 0x7f9f600b48a0 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f67c95640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600d1330 con 0x7f9f4c004800 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f65230640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f65230640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49760/0 (socket says 172.21.15.12:49760) 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.515+0000 7f9f65230640 1 -- 172.21.15.12:0/3209870083 learned_addr learned my addr 172.21.15.12:0/3209870083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 -- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 msgr2=0x7f9f600c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65a31640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 0x7f9f600c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f66232640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 0x7f9f600b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 0x7f9f600c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 -- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 msgr2=0x7f9f600b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 0x7f9f600b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 -- 172.21.15.12:0/3209870083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600b1e80 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65a31640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 0x7f9f600c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f66232640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 0x7f9f600b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f65230640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c000c00 tx=0x7f9f5c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c041070 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c005480 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f600b2120 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c001370 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.516+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f600b26e0 con 0x7f9f601587c0 2024-03-08T08:29:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.517+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f600749c0 con 0x7f9f601587c0 2024-03-08T08:29:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.518+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f5c044070 con 0x7f9f601587c0 2024-03-08T08:29:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.518+0000 7f9f527fc640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 0x7f9f4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.518+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f5c0abc40 con 0x7f9f601587c0 2024-03-08T08:29:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.518+0000 7f9f65a31640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 0x7f9f4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.518+0000 7f9f65a31640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 0x7f9f4406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f60073a40 tx=0x7f9f54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.522+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f5c07abd0 con 0x7f9f601587c0 2024-03-08T08:29:44.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.702+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f6006cbe0 con 0x7f9f601587c0 2024-03-08T08:29:44.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.703+0000 7f9f527fc640 1 -- 172.21.15.12:0/3209870083 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9f5c07a4d0 con 0x7f9f601587c0 2024-03-08T08:29:44.704 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:44.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 msgr2=0x7f9f4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 0x7f9f4406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f60073a40 tx=0x7f9f54009040 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 msgr2=0x7f9f60158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c000c00 tx=0x7f9f5c001120 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 shutdown_connections 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9f4406b4d0 0x7f9f4406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f600b48a0 0x7f9f600b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f601587c0 0x7f9f60158c40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 --2- 172.21.15.12:0/3209870083 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f4c004800 0x7f9f600c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 >> 172.21.15.12:0/3209870083 conn(0x7f9f600b3460 msgr2=0x7f9f60159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 shutdown_connections 2024-03-08T08:29:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.705+0000 7f9f67c95640 1 -- 172.21.15.12:0/3209870083 wait complete. 2024-03-08T08:29:44.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.860+0000 7f3b3b7f4640 1 Processor -- start 2024-03-08T08:29:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.860+0000 7f3b3b7f4640 1 -- start start 2024-03-08T08:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b3b7f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b340b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b3b7f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340d1d90 con 0x7f3b340b6b30 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b3b7f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340d2070 con 0x7f3b34161260 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b3b7f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b9ac0 con 0x7f3b3406c870 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b38d8f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3b340b6b30 0x7f3b3406c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44458/0 (socket says 172.21.15.12:44458) 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b38d8f640 1 -- 172.21.15.12:0/3099816911 learned_addr learned my addr 172.21.15.12:0/3099816911 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972571783 0 0) 0x7f3b340d1d90 con 0x7f3b340b6b30 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b39d91640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b340b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b20003520 con 0x7f3b340b6b30 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435566490 0 0) 0x7f3b20003520 con 0x7f3b340b6b30 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 msgr2=0x7f3b340b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b340b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 >> v1:172.21.15.156:6789/0 conn(0x7f3b34161260 legacy=0x7f3b340b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340d2350 con 0x7f3b340b6b30 2024-03-08T08:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3b24001670 con 0x7f3b340b6b30 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 >> v1:172.21.15.12:6789/0 conn(0x7f3b340b6b30 legacy=0x7f3b3406c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b200039e0 0x7f3b20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.861+0000 7f3b2bfff640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b20004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b2bfff640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b200050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000b9c0 con 0x7f3b20004390 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000bb70 con 0x7f3b200039e0 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000bd70 con 0x7f3b3406c870 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b20004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b39590640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b200039e0 0x7f3b20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b39d91640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b200050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 -- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 msgr2=0x7f3b200050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b3406c870 0x7f3b200050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 -- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b200039e0 msgr2=0x7f3b20003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b200039e0 0x7f3b20003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.862+0000 7f3b38d8f640 1 -- 172.21.15.12:0/3099816911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20003520 con 0x7f3b20004390 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b20004860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400c240 tx=0x7f3b20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa70ea628c96e3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24007d10 con 0x7f3b20004390 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24030070 con 0x7f3b20004390 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b2bfff640 1 -- 172.21.15.12:0/3099816911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24005cf0 con 0x7f3b20004390 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3099816911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 msgr2=0x7f3b20004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3099816911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b20004860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400c240 tx=0x7f3b20005830 comp rx=0 tx=0).stop 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b39d91640 1 -- 172.21.15.12:0/3099816911 reap_dead start 2024-03-08T08:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3099816911 shutdown_connections 2024-03-08T08:29:44.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3099816911 >> 172.21.15.12:0/3099816911 conn(0x7f3b34072930 msgr2=0x7f3b340d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:44.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3099816911 shutdown_connections 2024-03-08T08:29:44.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.863+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3099816911 wait complete. 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 Processor -- start 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 -- start start 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 0x7f3b340cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 0x7f3b340c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b6f90 con 0x7f3b20004390 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b2ec0 con 0x7f3b340c5ad0 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b3b7f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b7110 con 0x7f3b340cb520 2024-03-08T08:29:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b39590640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b39590640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49776/0 (socket says 172.21.15.12:49776) 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.865+0000 7f3b39590640 1 -- 172.21.15.12:0/3633719464 learned_addr learned my addr 172.21.15.12:0/3633719464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 0x7f3b340cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39d91640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 0x7f3b340c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 -- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 msgr2=0x7f3b340cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 0x7f3b340cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 -- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 msgr2=0x7f3b340c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 0x7f3b340c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 -- 172.21.15.12:0/3633719464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3414e7e0 con 0x7f3b20004390 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 0x7f3b340cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39d91640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 0x7f3b340c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b39590640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f7c0 tx=0x7f3b3000fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000fe70 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30018070 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b340cc090 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000d730 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.866+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b34148be0 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.867+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b340749c0 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.867+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b3000d8d0 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.868+0000 7f3b29ffb640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 0x7f3b1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.868+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b3008a240 con 0x7f3b20004390 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.868+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 0x7f3b1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.868+0000 7f3b38d8f640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 0x7f3b1806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b24034000 tx=0x7f3b24007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:44.872+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3b3001a310 con 0x7f3b20004390 2024-03-08T08:29:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.051+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b340b28f0 con 0x7f3b20004390 2024-03-08T08:29:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.051+0000 7f3b29ffb640 1 -- 172.21.15.12:0/3633719464 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b30059030 con 0x7f3b20004390 2024-03-08T08:29:45.052 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 msgr2=0x7f3b1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 0x7f3b1806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b24034000 tx=0x7f3b24007000 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 msgr2=0x7f3b340cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f7c0 tx=0x7f3b3000fc80 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 shutdown_connections 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3b1806b4d0 0x7f3b1806d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b340c5ad0 0x7f3b340c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b340cb520 0x7f3b340cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 --2- 172.21.15.12:0/3633719464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b20004390 0x7f3b340cd040 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 >> 172.21.15.12:0/3633719464 conn(0x7f3b34072930 msgr2=0x7f3b3415f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 shutdown_connections 2024-03-08T08:29:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.053+0000 7f3b3b7f4640 1 -- 172.21.15.12:0/3633719464 wait complete. 2024-03-08T08:29:45.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:45.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.166+0000 7fe377647640 1 Processor -- start 2024-03-08T08:29:45.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 -- start start 2024-03-08T08:29:45.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe37006cab0 0x7fe37009a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3700cfbb0 0x7fe3700d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3700d0150 con 0x7fe37015dda0 2024-03-08T08:29:45.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3700d0d40 con 0x7fe3700cfbb0 2024-03-08T08:29:45.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe377647640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3700d0410 con 0x7fe37006cab0 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe375be4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3700cfbb0 0x7fe3700d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe3753e3640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe37015dda0 0x7fe37006c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44464/0 (socket says 172.21.15.12:44464) 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.167+0000 7fe3753e3640 1 -- 172.21.15.12:0/519610664 learned_addr learned my addr 172.21.15.12:0/519610664 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.168+0000 7fe374be2640 1 --2- 172.21.15.12:0/519610664 >> v2:172.21.15.156:3300/0 conn(0x7fe37006cab0 0x7fe37009a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.168+0000 7fe375be4640 1 -- 172.21.15.12:0/519610664 >> v2:172.21.15.156:3300/0 conn(0x7fe37006cab0 msgr2=0x7fe37009a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.168+0000 7fe375be4640 1 --2- 172.21.15.12:0/519610664 >> v2:172.21.15.156:3300/0 conn(0x7fe37006cab0 0x7fe37009a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.168+0000 7fe375be4640 1 -- 172.21.15.12:0/519610664 >> v1:172.21.15.12:6789/0 conn(0x7fe37015dda0 legacy=0x7fe37006c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.168+0000 7fe375be4640 1 -- 172.21.15.12:0/519610664 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3700d06f0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe375be4640 1 --2- 172.21.15.12:0/519610664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe3700d11d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe37009aca0 tx=0x7fe36c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de82d92246f2e034 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe367fff640 1 -- 172.21.15.12:0/519610664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c039070 con 0x7fe3700cfbb0 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe367fff640 1 -- 172.21.15.12:0/519610664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c02cbf0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe367fff640 1 -- 172.21.15.12:0/519610664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c034440 con 0x7fe3700cfbb0 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 -- 172.21.15.12:0/519610664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 msgr2=0x7fe3700d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 --2- 172.21.15.12:0/519610664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe3700d11d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe37009aca0 tx=0x7fe36c02be40 comp rx=0 tx=0).stop 2024-03-08T08:29:45.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 -- 172.21.15.12:0/519610664 shutdown_connections 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 --2- 172.21.15.12:0/519610664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe3700d11d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 --2- 172.21.15.12:0/519610664 >> v2:172.21.15.156:3300/0 conn(0x7fe37006cab0 0x7fe37009a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 -- 172.21.15.12:0/519610664 >> 172.21.15.12:0/519610664 conn(0x7fe3700b3460 msgr2=0x7fe3700b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 -- 172.21.15.12:0/519610664 shutdown_connections 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.169+0000 7fe377647640 1 -- 172.21.15.12:0/519610664 wait complete. 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 Processor -- start 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 -- start start 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe37006cab0 0x7fe3700cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe37015dda0 0x7fe37015e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37006bf10 con 0x7fe3700cfbb0 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37015e180 con 0x7fe37015dda0 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe377647640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700cbfc0 con 0x7fe37006cab0 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe374be2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe374be2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49794/0 (socket says 172.21.15.12:49794) 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.171+0000 7fe374be2640 1 -- 172.21.15.12:0/3552073722 learned_addr learned my addr 172.21.15.12:0/3552073722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:45.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 -- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe37006cab0 msgr2=0x7fe3700cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe37006cab0 0x7fe3700cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe375be4640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe37015dda0 0x7fe37015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 -- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe37015dda0 msgr2=0x7fe37015e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe37015dda0 0x7fe37015e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 -- 172.21.15.12:0/3552073722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3700ca3b0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe374be2640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe36000e9e0 tx=0x7fe36000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360017070 con 0x7fe3700cfbb0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36000c9d0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3700c95f0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36000cdc0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.172+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3700c9bb0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.173+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3700b7080 con 0x7fe3700cfbb0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.174+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe36001a070 con 0x7fe3700cfbb0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.174+0000 7fe365ffb640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 0x7fe35406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.174+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe36008a2a0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.174+0000 7fe3753e3640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 0x7fe35406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.175+0000 7fe3753e3640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 0x7fe35406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe368000f60 tx=0x7fe368007a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.179+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe360059090 con 0x7fe3700cfbb0 2024-03-08T08:29:45.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.374+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe3700b78f0 con 0x7fe3700cfbb0 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.375+0000 7fe365ffb640 1 -- 172.21.15.12:0/3552073722 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe360058990 con 0x7fe3700cfbb0 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 msgr2=0x7fe35406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 0x7fe35406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe368000f60 tx=0x7fe368007a80 comp rx=0 tx=0).stop 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 msgr2=0x7fe37015c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe36000e9e0 tx=0x7fe36000eea0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 shutdown_connections 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe35406b4d0 0x7fe35406d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe37015dda0 0x7fe37015e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3700cfbb0 0x7fe37015c840 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 --2- 172.21.15.12:0/3552073722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe37006cab0 0x7fe3700cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 >> 172.21.15.12:0/3552073722 conn(0x7fe3700b3460 msgr2=0x7fe37006ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:45.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 shutdown_connections 2024-03-08T08:29:45.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.377+0000 7fe377647640 1 -- 172.21.15.12:0/3552073722 wait complete. 2024-03-08T08:29:45.389 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-08T08:29:45.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:45.389 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-08T08:29:45.423 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:45.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:29:45.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:45.423 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-08T08:29:45.483 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:45.483 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:45.498 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:45.498 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:45.552 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:45.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.694+0000 7fb333f81640 1 Processor -- start 2024-03-08T08:29:45.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.694+0000 7fb333f81640 1 -- start start 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb32c15ddc0 0x7fb32c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb32c0d0f40 0x7fb32c0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c0d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0b9030 con 0x7fb32c0d0f40 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0d1d20 con 0x7fb32c15ddc0 2024-03-08T08:29:45.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb333f81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c15e1a0 con 0x7fb32c0d26f0 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb32c0d0f40 0x7fb32c0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb32c0d0f40 0x7fb32c0d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49806/0 (socket says 172.21.15.12:49806) 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 -- 172.21.15.12:0/1091303163 learned_addr learned my addr 172.21.15.12:0/1091303163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33251e640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c0d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb331d1d640 1 --2- 172.21.15.12:0/1091303163 >> v2:172.21.15.156:3300/0 conn(0x7fb32c15ddc0 0x7fb32c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 -- 172.21.15.12:0/1091303163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 msgr2=0x7fb32c0d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c0d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 -- 172.21.15.12:0/1091303163 >> v2:172.21.15.156:3300/0 conn(0x7fb32c15ddc0 msgr2=0x7fb32c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 --2- 172.21.15.12:0/1091303163 >> v2:172.21.15.156:3300/0 conn(0x7fb32c15ddc0 0x7fb32c0d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33151c640 1 -- 172.21.15.12:0/1091303163 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0b8790 con 0x7fb32c0d0f40 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.695+0000 7fb33251e640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c0d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.696+0000 7fb33151c640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c0d21b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0b66a0 tx=0x7fb31c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e46e90969e4fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.696+0000 7fb330d1b640 1 -- 172.21.15.12:0/1091303163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c035020 con 0x7fb32c0d0f40 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.696+0000 7fb330d1b640 1 -- 172.21.15.12:0/1091303163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c031870 con 0x7fb32c0d0f40 2024-03-08T08:29:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.696+0000 7fb330d1b640 1 -- 172.21.15.12:0/1091303163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c031c40 con 0x7fb32c0d0f40 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 -- 172.21.15.12:0/1091303163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 msgr2=0x7fb32c0d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c0d21b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0b66a0 tx=0x7fb31c02b910 comp rx=0 tx=0).stop 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 -- 172.21.15.12:0/1091303163 shutdown_connections 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c0d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 --2- 172.21.15.12:0/1091303163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c0d21b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 --2- 172.21.15.12:0/1091303163 >> v2:172.21.15.156:3300/0 conn(0x7fb32c15ddc0 0x7fb32c0d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 -- 172.21.15.12:0/1091303163 >> 172.21.15.12:0/1091303163 conn(0x7fb32c072720 msgr2=0x7fb32c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 -- 172.21.15.12:0/1091303163 shutdown_connections 2024-03-08T08:29:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.697+0000 7fb333f81640 1 -- 172.21.15.12:0/1091303163 wait complete. 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.698+0000 7fb333f81640 1 Processor -- start 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.698+0000 7fb333f81640 1 -- start start 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 0x7fb32c0d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b9980 con 0x7fb32c0d0f40 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b8d50 con 0x7fb32c15ddc0 2024-03-08T08:29:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0d30c0 con 0x7fb32c0d26f0 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49812/0 (socket says 172.21.15.12:49812) 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 -- 172.21.15.12:0/3773115957 learned_addr learned my addr 172.21.15.12:0/3773115957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb33251e640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 0x7fb32c0d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb33151c640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 -- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 msgr2=0x7fb32c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c1580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 -- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 msgr2=0x7fb32c0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 0x7fb32c0d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 -- 172.21.15.12:0/3773115957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c1589c0 con 0x7fb32c0d0f40 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb33251e640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 0x7fb32c0d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb331d1d640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb32400d5f0 tx=0x7fb32400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240125f0 con 0x7fb32c0d0f40 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb324019070 con 0x7fb32c0d0f40 2024-03-08T08:29:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240129d0 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c155070 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.699+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c155630 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.700+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c0749c0 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.701+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb324012b70 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.701+0000 7fb3227fc640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 0x7fb31006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.701+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb32408a540 con 0x7fb32c0d0f40 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.701+0000 7fb33151c640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 0x7fb31006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.702+0000 7fb33151c640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 0x7fb31006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb31c036000 tx=0x7fb31c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.706+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb324059450 con 0x7fb32c0d0f40 2024-03-08T08:29:45.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.888+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb32c0cf870 con 0x7fb32c0d0f40 2024-03-08T08:29:45.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.889+0000 7fb3227fc640 1 -- 172.21.15.12:0/3773115957 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fb324058d50 con 0x7fb32c0d0f40 2024-03-08T08:29:45.890 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:45.890 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:45.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 msgr2=0x7fb31006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 0x7fb31006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb31c036000 tx=0x7fb31c03c040 comp rx=0 tx=0).stop 2024-03-08T08:29:45.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 msgr2=0x7fb32c160cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:45.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb32400d5f0 tx=0x7fb32400dab0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 shutdown_connections 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb31006b4d0 0x7fb31006d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb32c15ddc0 0x7fb32c0d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb32c0d26f0 0x7fb32c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 --2- 172.21.15.12:0/3773115957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c0d0f40 0x7fb32c160cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 >> 172.21.15.12:0/3773115957 conn(0x7fb32c072720 msgr2=0x7fb32c0b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 shutdown_connections 2024-03-08T08:29:45.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:45.891+0000 7fb333f81640 1 -- 172.21.15.12:0/3773115957 wait complete. 2024-03-08T08:29:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:45.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.006+0000 7f7475415640 1 Processor -- start 2024-03-08T08:29:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.006+0000 7f7475415640 1 -- start start 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f7475415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f74700cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f7475415640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747015f710 con 0x7f74701601d0 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f7475415640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747015cd40 con 0x7f74700cffb0 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f7475415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d0400 con 0x7f747015b950 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f746e7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f74701601d0 0x7f747015bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44476/0 (socket says 172.21.15.12:44476) 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.007+0000 7f746e7fc640 1 -- 172.21.15.12:0/3887998656 learned_addr learned my addr 172.21.15.12:0/3887998656 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f74700cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318971424 0 0) 0x7f747015f710 con 0x7f74701601d0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c003520 con 0x7f74701601d0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1214966099 0 0) 0x7f745c003520 con 0x7f74701601d0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 msgr2=0x7f74700cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f74700cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 >> v1:172.21.15.156:6789/0 conn(0x7f74700cffb0 legacy=0x7f747015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747015f9d0 con 0x7f74701601d0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f7458001670 con 0x7f74701601d0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 >> v1:172.21.15.12:6789/0 conn(0x7f74701601d0 legacy=0x7f747015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f745c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f745c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f745c004d40 0x7f745c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d0400 con 0x7f745c0039e0 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c00ba70 con 0x7f745c004d40 2024-03-08T08:29:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c00bc70 con 0x7f747015b950 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f745c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746e7fc640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f745c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746f7fe640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f745c004d40 0x7f745c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 -- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 msgr2=0x7f745c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f747015b950 0x7f745c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 -- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f745c004d40 msgr2=0x7f745c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f745c004d40 0x7f745c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.008+0000 7f746effd640 1 -- 172.21.15.12:0/3887998656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c003520 con 0x7f745c0039e0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.009+0000 7f746effd640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f745c003dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fc0 tx=0x7f745c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ed3d929117871b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.009+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460039070 con 0x7f745c0039e0 2024-03-08T08:29:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.009+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746002cc20 con 0x7f745c0039e0 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.009+0000 7f746dffb640 1 -- 172.21.15.12:0/3887998656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460009c10 con 0x7f745c0039e0 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 -- 172.21.15.12:0/3887998656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 msgr2=0x7f745c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 --2- 172.21.15.12:0/3887998656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f745c003dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fc0 tx=0x7f745c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f746f7fe640 1 -- 172.21.15.12:0/3887998656 reap_dead start 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 -- 172.21.15.12:0/3887998656 shutdown_connections 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 -- 172.21.15.12:0/3887998656 >> 172.21.15.12:0/3887998656 conn(0x7f7470099a40 msgr2=0x7f74700b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 -- 172.21.15.12:0/3887998656 shutdown_connections 2024-03-08T08:29:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.010+0000 7f7475415640 1 -- 172.21.15.12:0/3887998656 wait complete. 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.011+0000 7f7475415640 1 Processor -- start 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.011+0000 7f7475415640 1 -- start start 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74700c59d0 0x7f74700b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7470158c00 0x7f7470159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700cfbf0 con 0x7f745c0039e0 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700cfa70 con 0x7f7470158c00 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f7475415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700cfd70 con 0x7f74700c59d0 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49828/0 (socket says 172.21.15.12:49828) 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 -- 172.21.15.12:0/1766828053 learned_addr learned my addr 172.21.15.12:0/1766828053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746f7fe640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7470158c00 0x7f7470159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 -- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74700c59d0 msgr2=0x7f74700b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746e7fc640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74700c59d0 0x7f74700b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74700c59d0 0x7f74700b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 -- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7470158c00 msgr2=0x7f7470159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7470158c00 0x7f7470159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.012+0000 7f746effd640 1 -- 172.21.15.12:0/1766828053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74701595d0 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f746effd640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f746000e230 tx=0x7f7460009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460039050 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746003a040 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74700c5e50 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460003bb0 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.013+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74700c6400 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.014+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74700749c0 con 0x7f745c0039e0 2024-03-08T08:29:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.014+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7460004030 con 0x7f745c0039e0 2024-03-08T08:29:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.015+0000 7f74677fe640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 0x7f745006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.015+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74600ac9a0 con 0x7f745c0039e0 2024-03-08T08:29:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.015+0000 7f746e7fc640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 0x7f745006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.015+0000 7f746e7fc640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 0x7f745006d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7458001120 tx=0x7f7458009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.019+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f746007b940 con 0x7f745c0039e0 2024-03-08T08:29:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.198+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74700c8830 con 0x7f745c0039e0 2024-03-08T08:29:46.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.199+0000 7f74677fe640 1 -- 172.21.15.12:0/1766828053 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f746007b240 con 0x7f745c0039e0 2024-03-08T08:29:46.200 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 msgr2=0x7f745006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 0x7f745006d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7458001120 tx=0x7f7458009040 comp rx=0 tx=0).stop 2024-03-08T08:29:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 msgr2=0x7f747006efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f746000e230 tx=0x7f7460009750 comp rx=0 tx=0).stop 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 shutdown_connections 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f745006b500 0x7f745006d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7470158c00 0x7f7470159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74700c59d0 0x7f74700b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 --2- 172.21.15.12:0/1766828053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f745c0039e0 0x7f747006efd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 >> 172.21.15.12:0/1766828053 conn(0x7f7470099a40 msgr2=0x7f747015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 shutdown_connections 2024-03-08T08:29:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.201+0000 7f7475415640 1 -- 172.21.15.12:0/1766828053 wait complete. 2024-03-08T08:29:46.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:46.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.318+0000 7ff3b5f6c640 1 Processor -- start 2024-03-08T08:29:46.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.318+0000 7ff3b5f6c640 1 -- start start 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3b5f6c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3b015e0d0 0x7ff3b00cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3b5f6c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00d0290 con 0x7ff3b006bef0 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3b5f6c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00d0570 con 0x7ff3b0099b90 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3b5f6c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b00d0850 con 0x7ff3b015e0d0 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3aeffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3b015e0d0 0x7ff3b00cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3affff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff3b006bef0 0x7ff3b00cbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44482/0 (socket says 172.21.15.12:44482) 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.319+0000 7ff3affff640 1 -- 172.21.15.12:0/4247247385 learned_addr learned my addr 172.21.15.12:0/4247247385 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3ae7fc640 1 -- 172.21.15.12:0/4247247385 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128612980 0 0) 0x7ff3b00d0290 con 0x7ff3b006bef0 2024-03-08T08:29:46.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3ae7fc640 1 -- 172.21.15.12:0/4247247385 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff398003520 con 0x7ff3b006bef0 2024-03-08T08:29:46.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3aeffd640 1 -- 172.21.15.12:0/4247247385 >> v1:172.21.15.156:6789/0 conn(0x7ff3b0099b90 legacy=0x7ff3b0099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3aeffd640 1 -- 172.21.15.12:0/4247247385 >> v1:172.21.15.12:6789/0 conn(0x7ff3b006bef0 legacy=0x7ff3b00cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3aeffd640 1 -- 172.21.15.12:0/4247247385 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00d0b30 con 0x7ff3b015e0d0 2024-03-08T08:29:46.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3aeffd640 1 --2- 172.21.15.12:0/4247247385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b015e0d0 0x7ff3b00cfc40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40097a0 tx=0x7ff3a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3e2414c06d8cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3ae7fc640 1 -- 172.21.15.12:0/4247247385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4035020 con 0x7ff3b015e0d0 2024-03-08T08:29:46.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.320+0000 7ff3ae7fc640 1 -- 172.21.15.12:0/4247247385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a4031830 con 0x7ff3b015e0d0 2024-03-08T08:29:46.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3ae7fc640 1 -- 172.21.15.12:0/4247247385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4031c00 con 0x7ff3b015e0d0 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/4247247385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b015e0d0 msgr2=0x7ff3b00cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/4247247385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b015e0d0 0x7ff3b00cfc40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40097a0 tx=0x7ff3a402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/4247247385 shutdown_connections 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/4247247385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b015e0d0 0x7ff3b00cfc40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/4247247385 >> 172.21.15.12:0/4247247385 conn(0x7ff3b00b3460 msgr2=0x7ff3b00b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/4247247385 shutdown_connections 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.321+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/4247247385 wait complete. 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.322+0000 7ff3b5f6c640 1 Processor -- start 2024-03-08T08:29:46.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.322+0000 7ff3b5f6c640 1 -- start start 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3b006bef0 0x7ff3b015d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3b0099b90 0x7ff3b01585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b009b310 con 0x7ff3b0151120 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b009b190 con 0x7ff3b006bef0 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3b5f6c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00d1330 con 0x7ff3b0099b90 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49842/0 (socket says 172.21.15.12:49842) 2024-03-08T08:29:46.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 -- 172.21.15.12:0/393288505 learned_addr learned my addr 172.21.15.12:0/393288505 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 -- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3b0099b90 msgr2=0x7ff3b01585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3b0099b90 0x7ff3b01585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 -- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3b006bef0 msgr2=0x7ff3b015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3b006bef0 0x7ff3b015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.323+0000 7ff3affff640 1 -- 172.21.15.12:0/393288505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0158e40 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff3affff640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800cb40 tx=0x7ff3a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8017070 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a800aca0 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b00b47c0 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8012620 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.324+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b0155140 con 0x7ff3b0151120 2024-03-08T08:29:46.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.325+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b00749c0 con 0x7ff3b0151120 2024-03-08T08:29:46.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.325+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3a801a070 con 0x7ff3b0151120 2024-03-08T08:29:46.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.325+0000 7ff397fff640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 0x7ff39c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.326+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3a8089f50 con 0x7ff3b0151120 2024-03-08T08:29:46.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.326+0000 7ff3af7fe640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 0x7ff39c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.326+0000 7ff3af7fe640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 0x7ff39c06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000f60 tx=0x7ff3a00079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.330+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff3a8058e60 con 0x7ff3b0151120 2024-03-08T08:29:46.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.508+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3b006d500 con 0x7ff3b0151120 2024-03-08T08:29:46.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.509+0000 7ff397fff640 1 -- 172.21.15.12:0/393288505 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3a8058760 con 0x7ff3b0151120 2024-03-08T08:29:46.510 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:46.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 msgr2=0x7ff39c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 0x7ff39c06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000f60 tx=0x7ff3a00079d0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 msgr2=0x7ff3b0070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800cb40 tx=0x7ff3a800a060 comp rx=0 tx=0).stop 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 shutdown_connections 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ff39c06b500 0x7ff39c06d9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b0151120 0x7ff3b0070980 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3b0099b90 0x7ff3b01585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 --2- 172.21.15.12:0/393288505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3b006bef0 0x7ff3b015d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 >> 172.21.15.12:0/393288505 conn(0x7ff3b00b3460 msgr2=0x7ff3b006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 shutdown_connections 2024-03-08T08:29:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.511+0000 7ff3b5f6c640 1 -- 172.21.15.12:0/393288505 wait complete. 2024-03-08T08:29:46.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:46.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.626+0000 7fc824078640 1 Processor -- start 2024-03-08T08:29:46.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.626+0000 7fc824078640 1 -- start start 2024-03-08T08:29:46.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 0x7fc81c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c072c10 0x7fc81c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c0b5ac0 0x7fc81c06abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c0afd80 con 0x7fc81c072c10 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c0ad820 con 0x7fc81c15df00 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc824078640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0ada90 con 0x7fc81c0b5ac0 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c072c10 0x7fc81c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c072c10 0x7fc81c0b5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49850/0 (socket says 172.21.15.12:49850) 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 -- 172.21.15.12:0/3686282445 learned_addr learned my addr 172.21.15.12:0/3686282445 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 -- 172.21.15.12:0/3686282445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c0b5ac0 msgr2=0x7fc81c06abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 --2- 172.21.15.12:0/3686282445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c0b5ac0 0x7fc81c06abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821e14640 1 --2- 172.21.15.12:0/3686282445 >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 0x7fc81c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 -- 172.21.15.12:0/3686282445 >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 msgr2=0x7fc81c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 --2- 172.21.15.12:0/3686282445 >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 0x7fc81c0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821613640 1 -- 172.21.15.12:0/3686282445 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0add70 con 0x7fc81c072c10 2024-03-08T08:29:46.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.627+0000 7fc821e14640 1 --2- 172.21.15.12:0/3686282445 >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 0x7fc81c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.628+0000 7fc821613640 1 --2- 172.21.15.12:0/3686282445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c0b5580 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc81c072280 tx=0x7fc80c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b451033194114f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.628+0000 7fc820e12640 1 -- 172.21.15.12:0/3686282445 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c035020 con 0x7fc81c072c10 2024-03-08T08:29:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.628+0000 7fc820e12640 1 -- 172.21.15.12:0/3686282445 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c031870 con 0x7fc81c072c10 2024-03-08T08:29:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.628+0000 7fc820e12640 1 -- 172.21.15.12:0/3686282445 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c031c40 con 0x7fc81c072c10 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 -- 172.21.15.12:0/3686282445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 msgr2=0x7fc81c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 --2- 172.21.15.12:0/3686282445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c0b5580 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc81c072280 tx=0x7fc80c02b910 comp rx=0 tx=0).stop 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 -- 172.21.15.12:0/3686282445 shutdown_connections 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 --2- 172.21.15.12:0/3686282445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c0b5ac0 0x7fc81c06abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 --2- 172.21.15.12:0/3686282445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c0b5580 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 --2- 172.21.15.12:0/3686282445 >> v2:172.21.15.156:3300/0 conn(0x7fc81c15df00 0x7fc81c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 -- 172.21.15.12:0/3686282445 >> 172.21.15.12:0/3686282445 conn(0x7fc81c06c9d0 msgr2=0x7fc81c099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 -- 172.21.15.12:0/3686282445 shutdown_connections 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.629+0000 7fc824078640 1 -- 172.21.15.12:0/3686282445 wait complete. 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.630+0000 7fc824078640 1 Processor -- start 2024-03-08T08:29:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.630+0000 7fc824078640 1 -- start start 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 0x7fc81c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 0x7fc81c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c06b110 con 0x7fc81c072c10 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0ae330 con 0x7fc81c0b5ac0 2024-03-08T08:29:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c06b290 con 0x7fc81c15df00 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49862/0 (socket says 172.21.15.12:49862) 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 -- 172.21.15.12:0/2768759016 learned_addr learned my addr 172.21.15.12:0/2768759016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821613640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 0x7fc81c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc822615640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 0x7fc81c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 -- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 msgr2=0x7fc81c15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 0x7fc81c15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 -- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 msgr2=0x7fc81c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 0x7fc81c15f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 -- 172.21.15.12:0/2768759016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c145820 con 0x7fc81c072c10 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821613640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 0x7fc81c15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc822615640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 0x7fc81c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc821e14640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc81800c820 tx=0x7fc81800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818015070 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc818007a30 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c0c9d30 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818007de0 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.631+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c0ca2f0 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.632+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c0749c0 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.633+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc818018070 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.633+0000 7fc8127fc640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 0x7fc80006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.633+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc81808a130 con 0x7fc81c072c10 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.633+0000 7fc821613640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 0x7fc80006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.634+0000 7fc821613640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 0x7fc80006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc80c036000 tx=0x7fc80c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.637+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc81c145820 con 0x7fc81c072c10 2024-03-08T08:29:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.816+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc81c0d1670 con 0x7fc81c072c10 2024-03-08T08:29:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.816+0000 7fc8127fc640 1 -- 172.21.15.12:0/2768759016 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc818058fb0 con 0x7fc81c072c10 2024-03-08T08:29:46.817 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 msgr2=0x7fc80006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 0x7fc80006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc80c036000 tx=0x7fc80c008030 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 msgr2=0x7fc81c1452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc81800c820 tx=0x7fc81800cce0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 shutdown_connections 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc80006b4d0 0x7fc80006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc81c15df00 0x7fc81c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81c0b5ac0 0x7fc81c15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 --2- 172.21.15.12:0/2768759016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c072c10 0x7fc81c1452e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 >> 172.21.15.12:0/2768759016 conn(0x7fc81c06c9d0 msgr2=0x7fc81c0affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 shutdown_connections 2024-03-08T08:29:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.818+0000 7fc824078640 1 -- 172.21.15.12:0/2768759016 wait complete. 2024-03-08T08:29:46.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.931+0000 7efd0f8b4640 1 Processor -- start 2024-03-08T08:29:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.931+0000 7efd0f8b4640 1 -- start start 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0f8b4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efd0815ddc0 0x7efd080d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0f8b4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efd080b7bf0 0x7efd080b8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0f8b4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0815e1a0 con 0x7efd080d0f40 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0f8b4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080b9030 con 0x7efd0815ddc0 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0f8b4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080d1d20 con 0x7efd080b7bf0 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0d650640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efd0815ddc0 0x7efd080d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0ce4f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efd080d0f40 0x7efd080d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44498/0 (socket says 172.21.15.12:44498) 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0ce4f640 1 -- 172.21.15.12:0/521711776 learned_addr learned my addr 172.21.15.12:0/521711776 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0de51640 1 --2- 172.21.15.12:0/521711776 >> v2:172.21.15.156:3300/0 conn(0x7efd080b7bf0 0x7efd080b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0d650640 1 -- 172.21.15.12:0/521711776 >> v2:172.21.15.156:3300/0 conn(0x7efd080b7bf0 msgr2=0x7efd080b8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0d650640 1 --2- 172.21.15.12:0/521711776 >> v2:172.21.15.156:3300/0 conn(0x7efd080b7bf0 0x7efd080b8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0d650640 1 -- 172.21.15.12:0/521711776 >> v1:172.21.15.12:6789/0 conn(0x7efd080d0f40 legacy=0x7efd080d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efd0d650640 1 -- 172.21.15.12:0/521711776 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080b8790 con 0x7efd0815ddc0 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.932+0000 7efcfffff640 1 -- 172.21.15.12:0/521711776 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558146451 0 0) 0x7efd0815e1a0 con 0x7efd080d0f40 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0d650640 1 --2- 172.21.15.12:0/521711776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd0815ddc0 0x7efd080d0a00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efd080d0470 tx=0x7efd0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cecd6876e3fb5c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efcfffff640 1 -- 172.21.15.12:0/521711776 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04035020 con 0x7efd0815ddc0 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efcfffff640 1 -- 172.21.15.12:0/521711776 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04031830 con 0x7efd0815ddc0 2024-03-08T08:29:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efcfffff640 1 -- 172.21.15.12:0/521711776 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04031c00 con 0x7efd0815ddc0 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 -- 172.21.15.12:0/521711776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd0815ddc0 msgr2=0x7efd080d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/521711776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd0815ddc0 0x7efd080d0a00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efd080d0470 tx=0x7efd0402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 -- 172.21.15.12:0/521711776 shutdown_connections 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/521711776 >> v2:172.21.15.156:3300/0 conn(0x7efd080b7bf0 0x7efd080b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/521711776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd0815ddc0 0x7efd080d0a00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.933+0000 7efd0f8b4640 1 -- 172.21.15.12:0/521711776 >> 172.21.15.12:0/521711776 conn(0x7efd08072720 msgr2=0x7efd08072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.934+0000 7efd0f8b4640 1 -- 172.21.15.12:0/521711776 shutdown_connections 2024-03-08T08:29:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.934+0000 7efd0f8b4640 1 -- 172.21.15.12:0/521711776 wait complete. 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 Processor -- start 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 -- start start 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efd080d0f40 0x7efd0815e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd0815ddc0 0x7efd081606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd080b9980 con 0x7efd080b7bf0 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd080b8d50 con 0x7efd0815ddc0 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0f8b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd080d28c0 con 0x7efd080d0f40 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0d650640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0d650640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49876/0 (socket says 172.21.15.12:49876) 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.935+0000 7efd0d650640 1 -- 172.21.15.12:0/136990289 learned_addr learned my addr 172.21.15.12:0/136990289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0de51640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd0815ddc0 0x7efd081606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0ce4f640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efd080d0f40 0x7efd0815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 -- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efd080d0f40 msgr2=0x7efd0815e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efd080d0f40 0x7efd0815e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 -- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd0815ddc0 msgr2=0x7efd081606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd0815ddc0 0x7efd081606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 -- 172.21.15.12:0/136990289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080aa650 con 0x7efd080b7bf0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0d650640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efd04000c00 tx=0x7efd040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04040070 con 0x7efd080b7bf0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04039a20 con 0x7efd080b7bf0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd080aa930 con 0x7efd080b7bf0 2024-03-08T08:29:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04035040 con 0x7efd080b7bf0 2024-03-08T08:29:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.936+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd080aaef0 con 0x7efd080b7bf0 2024-03-08T08:29:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.937+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd080ab0f0 con 0x7efd080b7bf0 2024-03-08T08:29:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.938+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd0404b060 con 0x7efd080b7bf0 2024-03-08T08:29:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.938+0000 7efcfdffb640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.938+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd040aba40 con 0x7efd080b7bf0 2024-03-08T08:29:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.938+0000 7efd0ce4f640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.938+0000 7efd0ce4f640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 0x7efcec06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd081f1410 tx=0x7efcf8007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:46.942+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efd080ab0f0 con 0x7efd080b7bf0 2024-03-08T08:29:47.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.136+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efd080749c0 con 0x7efd080b7bf0 2024-03-08T08:29:47.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.136+0000 7efcfdffb640 1 -- 172.21.15.12:0/136990289 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7efd040abe20 con 0x7efd080b7bf0 2024-03-08T08:29:47.137 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.138+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 msgr2=0x7efcec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.138+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 0x7efcec06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd081f1410 tx=0x7efcf8007b50 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.138+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 msgr2=0x7efd0815bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.138+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efd04000c00 tx=0x7efd040046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.138+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 shutdown_connections 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd0815ddc0 0x7efd081606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efd080d0f40 0x7efd0815e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 --2- 172.21.15.12:0/136990289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd080b7bf0 0x7efd0815bc10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 >> 172.21.15.12:0/136990289 conn(0x7efd08072720 msgr2=0x7efd080c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 shutdown_connections 2024-03-08T08:29:47.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.139+0000 7efd0f8b4640 1 -- 172.21.15.12:0/136990289 wait complete. 2024-03-08T08:29:47.150 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:29:47.151 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:47.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:47.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 Processor -- start 2024-03-08T08:29:47.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 -- start start 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d00d1110 0x7f03d00b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d015e1a0 con 0x7f03d015ddc0 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d00b8790 con 0x7f03d00d31c0 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.305+0000 7f03d5bd7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b9030 con 0x7f03d00d1110 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03cf7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03d015ddc0 0x7f03d00d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44502/0 (socket says 172.21.15.12:44502) 2024-03-08T08:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03cf7fe640 1 -- 172.21.15.12:0/2305455802 learned_addr learned my addr 172.21.15.12:0/2305455802 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ceffd640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d00d1110 0x7f03d00b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052499211 0 0) 0x7f03d015e1a0 con 0x7f03d015ddc0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03bc003520 con 0x7f03d015ddc0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425519887 0 0) 0x7f03d00b8790 con 0x7f03d00d31c0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d015e1a0 con 0x7f03d00d31c0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472645358 0 0) 0x7f03bc003520 con 0x7f03d015ddc0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d00d1110 msgr2=0x7f03d00b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d00d1110 0x7f03d00b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 >> v1:172.21.15.156:6789/0 conn(0x7f03d00d31c0 legacy=0x7f03d00b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00b8a70 con 0x7f03d015ddc0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f03b8001670 con 0x7f03d015ddc0 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 >> v1:172.21.15.12:6789/0 conn(0x7f03d015ddc0 legacy=0x7f03d00d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03bc003b70 0x7f03bc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03bc004800 0x7f03bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc0051c0 0x7f03bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc00bfb0 con 0x7f03bc0051c0 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc00c160 con 0x7f03bc003b70 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc00c360 con 0x7f03bc004800 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03ceffd640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d00d1110 0x7f03d00b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03cffff640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc0051c0 0x7f03bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.306+0000 7f03cf7fe640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03bc003b70 0x7f03bc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03ceffd640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03bc004800 0x7f03bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 -- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03bc004800 msgr2=0x7f03bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03bc004800 0x7f03bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 -- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03bc003b70 msgr2=0x7f03bc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03bc003b70 0x7f03bc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 -- 172.21.15.12:0/2305455802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc003520 con 0x7f03bc0051c0 2024-03-08T08:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03cffff640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc0051c0 0x7f03bc00b9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f03c8001fc0 tx=0x7f03bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218ea4023129144c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8001be0 con 0x7f03bc0051c0 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c8003b90 con 0x7f03bc0051c0 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.307+0000 7f03ce7fc640 1 -- 172.21.15.12:0/2305455802 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80050d0 con 0x7f03bc0051c0 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 -- 172.21.15.12:0/2305455802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc0051c0 msgr2=0x7f03bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/2305455802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc0051c0 0x7f03bc00b9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f03c8001fc0 tx=0x7f03bc005830 comp rx=0 tx=0).stop 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03cffff640 1 -- 172.21.15.12:0/2305455802 reap_dead start 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 -- 172.21.15.12:0/2305455802 shutdown_connections 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 -- 172.21.15.12:0/2305455802 >> 172.21.15.12:0/2305455802 conn(0x7f03d0072720 msgr2=0x7f03d0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 -- 172.21.15.12:0/2305455802 shutdown_connections 2024-03-08T08:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.308+0000 7f03d5bd7640 1 -- 172.21.15.12:0/2305455802 wait complete. 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.309+0000 7f03d5bd7640 1 Processor -- start 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.309+0000 7f03d5bd7640 1 -- start start 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 0x7f03d0160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03d015f450 0x7f03d00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b9980 con 0x7f03d015f8d0 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00d1d20 con 0x7f03d015f450 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03d5bd7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b6be0 con 0x7f03d0152e40 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03cffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03cffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49900/0 (socket says 172.21.15.12:49900) 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03cffff640 1 -- 172.21.15.12:0/1117905917 learned_addr learned my addr 172.21.15.12:0/1117905917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03ceffd640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03d015f450 0x7f03d00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03cf7fe640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 0x7f03d0160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.310+0000 7f03cffff640 1 -- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 msgr2=0x7f03d0160f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cffff640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 0x7f03d0160f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cffff640 1 -- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03d015f450 msgr2=0x7f03d00cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cffff640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03d015f450 0x7f03d00cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cffff640 1 -- 172.21.15.12:0/1117905917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d015fea0 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cf7fe640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 0x7f03d0160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03cffff640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f03c80020b0 tx=0x7f03c8001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8043070 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c80050d0 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d0160180 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80054c0 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.311+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d0160740 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.312+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d00cfc70 con 0x7f03d015f8d0 2024-03-08T08:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.312+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03c8046070 con 0x7f03d015f8d0 2024-03-08T08:29:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.313+0000 7f03c7fff640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.313+0000 7f03cf7fe640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.313+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03c80aaf90 con 0x7f03d015f8d0 2024-03-08T08:29:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.313+0000 7f03cf7fe640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ca1a0 tx=0x7f03b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.317+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f03d00cfc70 con 0x7f03d015f8d0 2024-03-08T08:29:47.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.501+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d00749c0 con 0x7f03d015f8d0 2024-03-08T08:29:47.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.502+0000 7f03c7fff640 1 -- 172.21.15.12:0/1117905917 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f03c80a2e20 con 0x7f03d015f8d0 2024-03-08T08:29:47.503 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:47.503 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:47.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 msgr2=0x7f03b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ca1a0 tx=0x7f03b8009040 comp rx=0 tx=0).stop 2024-03-08T08:29:47.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 msgr2=0x7f03d00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f03c80020b0 tx=0x7f03c8001650 comp rx=0 tx=0).stop 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 shutdown_connections 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d015f8d0 0x7f03d00cc560 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03d015f450 0x7f03d00cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 --2- 172.21.15.12:0/1117905917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03d0152e40 0x7f03d0160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 >> 172.21.15.12:0/1117905917 conn(0x7f03d0072720 msgr2=0x7f03d00d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 shutdown_connections 2024-03-08T08:29:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.504+0000 7f03d5bd7640 1 -- 172.21.15.12:0/1117905917 wait complete. 2024-03-08T08:29:47.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:47.517 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.619+0000 7f1d8169a640 1 Processor -- start 2024-03-08T08:29:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.619+0000 7f1d8169a640 1 -- start start 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d8169a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1d7c0cc390 0x7f1d7c0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d8169a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0cdf60 con 0x7f1d7c099b90 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d8169a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0cf700 con 0x7f1d7c15bb90 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d8169a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cd590 con 0x7f1d7c0cc390 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d7b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1d7c099b90 0x7f1d7c0c53d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44514/0 (socket says 172.21.15.12:44514) 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/1583608409 learned_addr learned my addr 172.21.15.12:0/1583608409 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> v2:172.21.15.156:3300/0 conn(0x7f1d7c0cc390 0x7f1d7c0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918131344 0 0) 0x7f1d7c0cdf60 con 0x7f1d7c099b90 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.620+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d68003520 con 0x7f1d7c099b90 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616367771 0 0) 0x7f1d68003520 con 0x7f1d7c099b90 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 >> v2:172.21.15.156:3300/0 conn(0x7f1d7c0cc390 msgr2=0x7f1d7c0cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 --2- 172.21.15.12:0/1583608409 >> v2:172.21.15.156:3300/0 conn(0x7f1d7c0cc390 0x7f1d7c0cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 >> v1:172.21.15.156:6789/0 conn(0x7f1d7c15bb90 legacy=0x7f1d7c0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0cc980 con 0x7f1d7c099b90 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f1d740019f0 con 0x7f1d7c099b90 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 >> v1:172.21.15.12:6789/0 conn(0x7f1d7c099b90 legacy=0x7f1d7c0c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d680039b0 0x7f1d68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 0x7f1d68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d680051c0 0x7f1d6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cd590 con 0x7f1d68004800 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6800c060 con 0x7f1d680039b0 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6800c260 con 0x7f1d680051c0 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> v2:172.21.15.156:3300/0 conn(0x7f1d7c0cc390 0x7f1d7c0cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 0x7f1d68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7affd640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d680039b0 0x7f1d68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7b7fe640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d680051c0 0x7f1d6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 -- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d680051c0 msgr2=0x7f1d6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d680051c0 0x7f1d6800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 -- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d680039b0 msgr2=0x7f1d68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d680039b0 0x7f1d68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d727fc640 1 -- 172.21.15.12:0/1583608409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d68003520 con 0x7f1d68004800 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.621+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/1583608409 reap_dead start 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.622+0000 7f1d727fc640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 0x7f1d68004c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009c70 tx=0x7f1d68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5628f063db02091 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.622+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64009840 con 0x7f1d68004800 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.622+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64004020 con 0x7f1d68004800 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.622+0000 7f1d7a7fc640 1 -- 172.21.15.12:0/1583608409 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64004450 con 0x7f1d68004800 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 -- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 msgr2=0x7f1d68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 0x7f1d68004c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009c70 tx=0x7f1d68005830 comp rx=0 tx=0).stop 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 -- 172.21.15.12:0/1583608409 shutdown_connections 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 --2- 172.21.15.12:0/1583608409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d68004800 0x7f1d68004c80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 -- 172.21.15.12:0/1583608409 >> 172.21.15.12:0/1583608409 conn(0x7f1d7c0b3370 msgr2=0x7f1d7c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 -- 172.21.15.12:0/1583608409 shutdown_connections 2024-03-08T08:29:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.623+0000 7f1d8169a640 1 -- 172.21.15.12:0/1583608409 wait complete. 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.624+0000 7f1d8169a640 1 Processor -- start 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.624+0000 7f1d8169a640 1 -- start start 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 0x7f1d7c14eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 0x7f1d7c0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cd220 con 0x7f1d7c0cb540 2024-03-08T08:29:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0ccf40 con 0x7f1d68004800 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c15bf70 con 0x7f1d7c0c9980 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49930/0 (socket says 172.21.15.12:49930) 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/3392115839 learned_addr learned my addr 172.21.15.12:0/3392115839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7affd640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 0x7f1d7c14eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d727fc640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 0x7f1d7c0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 msgr2=0x7f1d7c0cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 0x7f1d7c0cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 msgr2=0x7f1d7c14eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 0x7f1d7c14eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 -- 172.21.15.12:0/3392115839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0cbb10 con 0x7f1d7c0cb540 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7affd640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 0x7f1d7c14eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d727fc640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 0x7f1d7c0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d7b7fe640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0b3f40 tx=0x7f1d74007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7400ea40 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74018070 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c0b5810 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7400ee00 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.625+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c14f0e0 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.626+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0749c0 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.627+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d7401d460 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.627+0000 7f1d73fff640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 0x7f1d5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.627+0000 7f1d7affd640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 0x7f1d5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.627+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d7408a490 con 0x7f1d7c0cb540 2024-03-08T08:29:47.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.628+0000 7f1d7affd640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 0x7f1d5c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c073a40 tx=0x7f1d6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.632+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1d74059300 con 0x7f1d7c0cb540 2024-03-08T08:29:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.810+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d7c070080 con 0x7f1d7c0cb540 2024-03-08T08:29:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.810+0000 7f1d73fff640 1 -- 172.21.15.12:0/3392115839 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d74058c00 con 0x7f1d7c0cb540 2024-03-08T08:29:47.811 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:47.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 msgr2=0x7f1d5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 0x7f1d5c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c073a40 tx=0x7f1d6c009040 comp rx=0 tx=0).stop 2024-03-08T08:29:47.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 msgr2=0x7f1d7c0cf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0b3f40 tx=0x7f1d74007af0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 shutdown_connections 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1d5c06b4d0 0x7f1d5c06d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d7c0cb540 0x7f1d7c0cf930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d7c0c9980 0x7f1d7c0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 --2- 172.21.15.12:0/3392115839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d68004800 0x7f1d7c14eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 >> 172.21.15.12:0/3392115839 conn(0x7f1d7c0b3370 msgr2=0x7f1d7c0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 shutdown_connections 2024-03-08T08:29:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.812+0000 7f1d8169a640 1 -- 172.21.15.12:0/3392115839 wait complete. 2024-03-08T08:29:47.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.927+0000 7f07ba655640 1 Processor -- start 2024-03-08T08:29:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.927+0000 7f07ba655640 1 -- start start 2024-03-08T08:29:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07ba655640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07b415dec0 0x7f07b4161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07ba655640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 0x7f07b40d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07ba655640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b40b6be0 con 0x7f07b40d0790 2024-03-08T08:29:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07ba655640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b40d2140 con 0x7f07b415dec0 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07ba655640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b415e2a0 con 0x7f07b4161ed0 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07b415dec0 0x7f07b4161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07b415dec0 0x7f07b4161990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49944/0 (socket says 172.21.15.12:49944) 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 -- 172.21.15.12:0/2781298484 learned_addr learned my addr 172.21.15.12:0/2781298484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b37fe640 1 --2- 172.21.15.12:0/2781298484 >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 0x7f07b40d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 -- 172.21.15.12:0/2781298484 >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 msgr2=0x7f07b40d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 --2- 172.21.15.12:0/2781298484 >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 0x7f07b40d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 -- 172.21.15.12:0/2781298484 >> v1:172.21.15.156:6789/0 conn(0x7f07b40d0790 legacy=0x7f07b40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b3fff640 1 -- 172.21.15.12:0/2781298484 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b4162350 con 0x7f07b415dec0 2024-03-08T08:29:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.928+0000 7f07b37fe640 1 --2- 172.21.15.12:0/2781298484 >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 0x7f07b40d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:47.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.929+0000 7f07b3fff640 1 --2- 172.21.15.12:0/2781298484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b415dec0 0x7f07b4161990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07b408ca20 tx=0x7f07a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a49ea31ec37b242 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.929+0000 7f07b2ffd640 1 -- 172.21.15.12:0/2781298484 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8035020 con 0x7f07b415dec0 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.929+0000 7f07b2ffd640 1 -- 172.21.15.12:0/2781298484 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a8031870 con 0x7f07b415dec0 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.929+0000 7f07b2ffd640 1 -- 172.21.15.12:0/2781298484 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8031c40 con 0x7f07b415dec0 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 -- 172.21.15.12:0/2781298484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b415dec0 msgr2=0x7f07b4161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 --2- 172.21.15.12:0/2781298484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b415dec0 0x7f07b4161990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07b408ca20 tx=0x7f07a802b910 comp rx=0 tx=0).stop 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 -- 172.21.15.12:0/2781298484 shutdown_connections 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 --2- 172.21.15.12:0/2781298484 >> v2:172.21.15.156:3300/0 conn(0x7f07b4161ed0 0x7f07b40d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 --2- 172.21.15.12:0/2781298484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b415dec0 0x7f07b4161990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 -- 172.21.15.12:0/2781298484 >> 172.21.15.12:0/2781298484 conn(0x7f07b406c240 msgr2=0x7f07b406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 -- 172.21.15.12:0/2781298484 shutdown_connections 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.930+0000 7f07ba655640 1 -- 172.21.15.12:0/2781298484 wait complete. 2024-03-08T08:29:47.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.931+0000 7f07ba655640 1 Processor -- start 2024-03-08T08:29:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.931+0000 7f07ba655640 1 -- start start 2024-03-08T08:29:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07b40d0790 0x7f07b40c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07b415dec0 0x7f07b41584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40d0bd0 con 0x7f07b4161ed0 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40d1dd0 con 0x7f07b415dec0 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40b94f0 con 0x7f07b40d0790 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49956/0 (socket says 172.21.15.12:49956) 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 -- 172.21.15.12:0/3938786118 learned_addr learned my addr 172.21.15.12:0/3938786118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b37fe640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07b415dec0 0x7f07b41584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b3fff640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07b40d0790 0x7f07b40c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 -- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07b40d0790 msgr2=0x7f07b40c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07b40d0790 0x7f07b40c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 -- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07b415dec0 msgr2=0x7f07b41584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07b415dec0 0x7f07b41584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 -- 172.21.15.12:0/3938786118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b415c5d0 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b8bf2640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c7b0 tx=0x7f07ac00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00ce60 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac004b10 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b40c8190 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00a850 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.932+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b40c8750 con 0x7f07b4161ed0 2024-03-08T08:29:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.934+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b40749c0 con 0x7f07b4161ed0 2024-03-08T08:29:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.934+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07ac00a9f0 con 0x7f07b4161ed0 2024-03-08T08:29:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.934+0000 7f07b0ff9640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 0x7f07a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.934+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07ac08a530 con 0x7f07b4161ed0 2024-03-08T08:29:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.934+0000 7f07b3fff640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 0x7f07a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.935+0000 7f07b3fff640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 0x7f07a006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07a8036000 tx=0x7f07a803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:47.939+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f07b415c5d0 con 0x7f07b4161ed0 2024-03-08T08:29:48.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.121+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07b40b74b0 con 0x7f07b4161ed0 2024-03-08T08:29:48.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.122+0000 7f07b0ff9640 1 -- 172.21.15.12:0/3938786118 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07ac058d60 con 0x7f07b4161ed0 2024-03-08T08:29:48.123 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 msgr2=0x7f07a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 0x7f07a006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07a8036000 tx=0x7f07a803c040 comp rx=0 tx=0).stop 2024-03-08T08:29:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 msgr2=0x7f07b40b8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c7b0 tx=0x7f07ac00cc70 comp rx=0 tx=0).stop 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 shutdown_connections 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f07a006b4d0 0x7f07a006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07b4161ed0 0x7f07b40b8190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07b415dec0 0x7f07b41584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 --2- 172.21.15.12:0/3938786118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07b40d0790 0x7f07b40c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 >> 172.21.15.12:0/3938786118 conn(0x7f07b406c240 msgr2=0x7f07b40d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 shutdown_connections 2024-03-08T08:29:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.124+0000 7f07ba655640 1 -- 172.21.15.12:0/3938786118 wait complete. 2024-03-08T08:29:48.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.239+0000 7fa63f683640 1 Processor -- start 2024-03-08T08:29:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.239+0000 7fa63f683640 1 -- start start 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63f683640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6380d0250 0x7fa6380d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63f683640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa6380d25d0 0x7fa6380d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63f683640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6380b6be0 con 0x7fa63815dec0 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63f683640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6380d2140 con 0x7fa6380d0250 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63f683640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63815e2a0 con 0x7fa6380d25d0 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6380d0250 0x7fa6380d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6380d0250 0x7fa6380d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49966/0 (socket says 172.21.15.12:49966) 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63d41f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa63815dec0 0x7fa638161990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44526/0 (socket says 172.21.15.12:44526) 2024-03-08T08:29:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 -- 172.21.15.12:0/329335516 learned_addr learned my addr 172.21.15.12:0/329335516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:48.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 -- 172.21.15.12:0/329335516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa6380d25d0 msgr2=0x7fa6380d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:48.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 --2- 172.21.15.12:0/329335516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa6380d25d0 0x7fa6380d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa637fff640 1 -- 172.21.15.12:0/329335516 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055480943 0 0) 0x7fa630001110 con 0x7fa63815dec0 2024-03-08T08:29:48.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 -- 172.21.15.12:0/329335516 >> v1:172.21.15.12:6789/0 conn(0x7fa63815dec0 legacy=0x7fa638161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.240+0000 7fa63cc1e640 1 -- 172.21.15.12:0/329335516 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6381620a0 con 0x7fa6380d0250 2024-03-08T08:29:48.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.241+0000 7fa63cc1e640 1 --2- 172.21.15.12:0/329335516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa6380d06b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b78095f5d80ff1f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.241+0000 7fa637fff640 1 -- 172.21.15.12:0/329335516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c034070 con 0x7fa6380d0250 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.241+0000 7fa637fff640 1 -- 172.21.15.12:0/329335516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c030840 con 0x7fa6380d0250 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.241+0000 7fa637fff640 1 -- 172.21.15.12:0/329335516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c030c10 con 0x7fa6380d0250 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 -- 172.21.15.12:0/329335516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 msgr2=0x7fa6380d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 --2- 172.21.15.12:0/329335516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa6380d06b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b980 comp rx=0 tx=0).stop 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 -- 172.21.15.12:0/329335516 shutdown_connections 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 --2- 172.21.15.12:0/329335516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa6380d25d0 0x7fa6380d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 --2- 172.21.15.12:0/329335516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa6380d06b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 -- 172.21.15.12:0/329335516 >> 172.21.15.12:0/329335516 conn(0x7fa63806c240 msgr2=0x7fa63806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 -- 172.21.15.12:0/329335516 shutdown_connections 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.242+0000 7fa63f683640 1 -- 172.21.15.12:0/329335516 wait complete. 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.243+0000 7fa63f683640 1 Processor -- start 2024-03-08T08:29:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.243+0000 7fa63f683640 1 -- start start 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6380d25d0 0x7fa63815e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 0x7fa6380cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638162630 con 0x7fa6380d0250 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638162350 con 0x7fa6380d25d0 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63f683640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380d1dd0 con 0x7fa63815dec0 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49974/0 (socket says 172.21.15.12:49974) 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 -- 172.21.15.12:0/2504474627 learned_addr learned my addr 172.21.15.12:0/2504474627 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63cc1e640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6380d25d0 0x7fa63815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63dc20640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 0x7fa6380cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 -- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 msgr2=0x7fa6380cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 0x7fa6380cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 -- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6380d25d0 msgr2=0x7fa63815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6380d25d0 0x7fa63815e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 -- 172.21.15.12:0/2504474627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6381e7870 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63dc20640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 0x7fa6380cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.244+0000 7fa63d41f640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa63000c410 tx=0x7fa63000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.245+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300074e0 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.245+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa630007680 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.245+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6381e7b50 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.245+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630004a00 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.245+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6381e8110 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.246+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6380749c0 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.246+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa63001a020 con 0x7fa6380d0250 2024-03-08T08:29:48.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.246+0000 7fa635ffb640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 0x7fa62406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.247+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6300899c0 con 0x7fa6380d0250 2024-03-08T08:29:48.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.247+0000 7fa63cc1e640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 0x7fa62406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.247+0000 7fa63cc1e640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 0x7fa62406d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.251+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa63008be20 con 0x7fa6380d0250 2024-03-08T08:29:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.429+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa6380d1750 con 0x7fa6380d0250 2024-03-08T08:29:48.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.429+0000 7fa635ffb640 1 -- 172.21.15.12:0/2504474627 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa6380b6be0 con 0x7fa6380d0250 2024-03-08T08:29:48.431 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 msgr2=0x7fa62406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 0x7fa62406d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c008030 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 msgr2=0x7fa63814be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa63000c410 tx=0x7fa63000c8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 shutdown_connections 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa62406b4d0 0x7fa62406d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa63815dec0 0x7fa6380cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6380d25d0 0x7fa63815e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 --2- 172.21.15.12:0/2504474627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6380d0250 0x7fa63814be40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.431+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 >> 172.21.15.12:0/2504474627 conn(0x7fa63806c240 msgr2=0x7fa6380b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.432+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 shutdown_connections 2024-03-08T08:29:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.432+0000 7fa63f683640 1 -- 172.21.15.12:0/2504474627 wait complete. 2024-03-08T08:29:48.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.545+0000 7f2c11ecd640 1 Processor -- start 2024-03-08T08:29:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.545+0000 7f2c11ecd640 1 -- start start 2024-03-08T08:29:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.545+0000 7f2c11ecd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c06c870 0x7f2c0c0d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.545+0000 7f2c11ecd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c11ecd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0d1840 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c11ecd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0d25f0 con 0x7f2c0c0b9440 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c11ecd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0d28d0 con 0x7f2c0c06c870 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51676/0 (socket says 172.21.15.12:51676) 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 -- 172.21.15.12:0/3183114349 learned_addr learned my addr 172.21.15.12:0/3183114349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0affd640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c06c870 0x7f2c0c0d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 -- 172.21.15.12:0/3183114349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c06c870 msgr2=0x7f2c0c0d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c06c870 0x7f2c0c0d1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 -- 172.21.15.12:0/3183114349 >> v1:172.21.15.156:6789/0 conn(0x7f2c0c15dbd0 legacy=0x7f2c0c06c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.546+0000 7f2c0bfff640 1 -- 172.21.15.12:0/3183114349 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0d2bb0 con 0x7f2c0c0b9440 2024-03-08T08:29:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0c7420 tx=0x7f2c0402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8566fc964342d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c0a7fc640 1 -- 172.21.15.12:0/3183114349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0402ca30 con 0x7f2c0c0b9440 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c0a7fc640 1 -- 172.21.15.12:0/3183114349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0402cbd0 con 0x7f2c0c0b9440 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c0a7fc640 1 -- 172.21.15.12:0/3183114349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c040343f0 con 0x7f2c0c0b9440 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 -- 172.21.15.12:0/3183114349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c0b9440 msgr2=0x7f2c0c0b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0c7420 tx=0x7f2c0402be90 comp rx=0 tx=0).stop 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 -- 172.21.15.12:0/3183114349 shutdown_connections 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c0b9440 0x7f2c0c0b6bc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/3183114349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c06c870 0x7f2c0c0d1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 -- 172.21.15.12:0/3183114349 >> 172.21.15.12:0/3183114349 conn(0x7f2c0c072930 msgr2=0x7f2c0c0d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 -- 172.21.15.12:0/3183114349 shutdown_connections 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.547+0000 7f2c11ecd640 1 -- 172.21.15.12:0/3183114349 wait complete. 2024-03-08T08:29:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 Processor -- start 2024-03-08T08:29:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 -- start start 2024-03-08T08:29:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c0c06c870 0x7f2c0c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c0b9440 0x7f2c0c153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0d3690 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0d3170 con 0x7f2c0c06c870 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.549+0000 7f2c11ecd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0d3810 con 0x7f2c0c0b9440 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51684/0 (socket says 172.21.15.12:51684) 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 -- 172.21.15.12:0/4135406275 learned_addr learned my addr 172.21.15.12:0/4135406275 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0affd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c0b9440 0x7f2c0c153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c0c06c870 0x7f2c0c156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 -- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c0b9440 msgr2=0x7f2c0c153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c0b9440 0x7f2c0c153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 -- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c0c06c870 msgr2=0x7f2c0c156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c0c06c870 0x7f2c0c156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 -- 172.21.15.12:0/4135406275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c155f30 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2c04036040 tx=0x7f2c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c04043020 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0403e450 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0403e860 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c156210 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.550+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c1567d0 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.551+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c0749c0 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.552+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c04046020 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.552+0000 7f2bf3fff640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 0x7f2bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:48.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.552+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c040acb70 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.552+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 0x7f2bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:48.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.553+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 0x7f2bf806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00a750 tx=0x7f2bfc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:48.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.557+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2c0404d030 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.749+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2c0c0b7a70 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.750+0000 7f2bf3fff640 1 -- 172.21.15.12:0/4135406275 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2c0407b9f0 con 0x7f2c0c15dbd0 2024-03-08T08:29:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 msgr2=0x7f2bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 0x7f2bf806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00a750 tx=0x7f2bfc008040 comp rx=0 tx=0).stop 2024-03-08T08:29:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 msgr2=0x7f2c0c1e8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2c04036040 tx=0x7f2c04004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 shutdown_connections 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f2bf806b4d0 0x7f2bf806d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c15dbd0 0x7f2c0c1e8700 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2c0c0b9440 0x7f2c0c153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 --2- 172.21.15.12:0/4135406275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c0c06c870 0x7f2c0c156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 >> 172.21.15.12:0/4135406275 conn(0x7f2c0c072930 msgr2=0x7f2c0c15a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 shutdown_connections 2024-03-08T08:29:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:48.752+0000 7f2c11ecd640 1 -- 172.21.15.12:0/4135406275 wait complete. 2024-03-08T08:29:48.764 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-03-08T08:29:48.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:48.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-08T08:29:48.790 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:48.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:29:48.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:48.791 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-08T08:29:48.845 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:48.846 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:48.864 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:48.865 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:48.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:49.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.023+0000 7f1cf2064640 1 Processor -- start 2024-03-08T08:29:49.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.023+0000 7f1cf2064640 1 -- start start 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cf2064640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1cec15dda0 0x7f1cec09a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cf2064640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cec06ce00 0x7f1cec0d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cf2064640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec06bef0 con 0x7f1cec0b2ae0 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cf2064640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec06c1d0 con 0x7f1cec06ce00 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cf2064640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec0cff70 con 0x7f1cec15dda0 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cebfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cec06ce00 0x7f1cec0d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cebfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1cec06ce00 0x7f1cec0d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51692/0 (socket says 172.21.15.12:51692) 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.024+0000 7f1cebfff640 1 -- 172.21.15.12:0/706982538 learned_addr learned my addr 172.21.15.12:0/706982538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cebfff640 1 -- 172.21.15.12:0/706982538 >> v2:172.21.15.156:3300/0 conn(0x7f1cec15dda0 msgr2=0x7f1cec09a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cebfff640 1 --2- 172.21.15.12:0/706982538 >> v2:172.21.15.156:3300/0 conn(0x7f1cec15dda0 0x7f1cec09a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cebfff640 1 -- 172.21.15.12:0/706982538 >> v1:172.21.15.156:6789/0 conn(0x7f1cec0b2ae0 legacy=0x7f1cec06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cebfff640 1 -- 172.21.15.12:0/706982538 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0d0250 con 0x7f1cec06ce00 2024-03-08T08:29:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cebfff640 1 --2- 172.21.15.12:0/706982538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec06ce00 0x7f1cec0d0770 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06f5f0 tx=0x7f1ce402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6084272b88d7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.025+0000 7f1cea7fc640 1 -- 172.21.15.12:0/706982538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce402ca80 con 0x7f1cec06ce00 2024-03-08T08:29:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cea7fc640 1 -- 172.21.15.12:0/706982538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce402cc20 con 0x7f1cec06ce00 2024-03-08T08:29:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cea7fc640 1 -- 172.21.15.12:0/706982538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40343f0 con 0x7f1cec06ce00 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 -- 172.21.15.12:0/706982538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec06ce00 msgr2=0x7f1cec0d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 --2- 172.21.15.12:0/706982538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec06ce00 0x7f1cec0d0770 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06f5f0 tx=0x7f1ce402bee0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 -- 172.21.15.12:0/706982538 shutdown_connections 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 --2- 172.21.15.12:0/706982538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec06ce00 0x7f1cec0d0770 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 --2- 172.21.15.12:0/706982538 >> v2:172.21.15.156:3300/0 conn(0x7f1cec15dda0 0x7f1cec09a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 -- 172.21.15.12:0/706982538 >> 172.21.15.12:0/706982538 conn(0x7f1cec0b3460 msgr2=0x7f1cec0b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 -- 172.21.15.12:0/706982538 shutdown_connections 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.026+0000 7f1cf2064640 1 -- 172.21.15.12:0/706982538 wait complete. 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.027+0000 7f1cf2064640 1 Processor -- start 2024-03-08T08:29:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.027+0000 7f1cf2064640 1 -- start start 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cec06ce00 0x7f1cec153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cec0b2ae0 0x7f1cec157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0cfd30 con 0x7f1cec15dda0 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0cfbb0 con 0x7f1cec0b2ae0 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cf2064640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0cbfc0 con 0x7f1cec06ce00 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51704/0 (socket says 172.21.15.12:51704) 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 -- 172.21.15.12:0/292351372 learned_addr learned my addr 172.21.15.12:0/292351372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 -- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cec06ce00 msgr2=0x7f1cec153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cec06ce00 0x7f1cec153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 -- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cec0b2ae0 msgr2=0x7f1cec157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cec0b2ae0 0x7f1cec157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.028+0000 7f1cebfff640 1 -- 172.21.15.12:0/292351372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec157ba0 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cebfff640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009f40 tx=0x7f1ce4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4043020 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce403e450 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec149190 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce403e820 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.029+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec158120 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.030+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec0749c0 con 0x7f1cec15dda0 2024-03-08T08:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.030+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ce4046020 con 0x7f1cec15dda0 2024-03-08T08:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.031+0000 7f1cd3fff640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 0x7f1cd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.031+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ce40acb70 con 0x7f1cec15dda0 2024-03-08T08:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.031+0000 7f1ceb7fe640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 0x7f1cd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.031+0000 7f1ceb7fe640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 0x7f1cd806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009710 tx=0x7f1cdc006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.035+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ce404d030 con 0x7f1cec15dda0 2024-03-08T08:29:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.214+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cec06c4d0 con 0x7f1cec15dda0 2024-03-08T08:29:49.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.215+0000 7f1cd3fff640 1 -- 172.21.15.12:0/292351372 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f1ce4098100 con 0x7f1cec15dda0 2024-03-08T08:29:49.216 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:49.217 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 msgr2=0x7f1cd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 0x7f1cd806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009710 tx=0x7f1cdc006c90 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 msgr2=0x7f1cec148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009f40 tx=0x7f1ce4004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 shutdown_connections 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1cd806b4d0 0x7f1cd806d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cec15dda0 0x7f1cec148c50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cec0b2ae0 0x7f1cec157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 --2- 172.21.15.12:0/292351372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1cec06ce00 0x7f1cec153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 >> 172.21.15.12:0/292351372 conn(0x7f1cec0b3460 msgr2=0x7f1cec15c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 shutdown_connections 2024-03-08T08:29:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.217+0000 7f1cf2064640 1 -- 172.21.15.12:0/292351372 wait complete. 2024-03-08T08:29:49.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:49.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:49.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.332+0000 7f657a1c7640 1 Processor -- start 2024-03-08T08:29:49.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 -- start start 2024-03-08T08:29:49.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f657415e150 0x7f657406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f657406cb10 0x7f65740cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657415e7b0 con 0x7f657415dd70 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657406cf80 con 0x7f657415e150 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f657a1c7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657415ea90 con 0x7f657406cb10 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f6572ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f657415e150 0x7f657406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.333+0000 7f65737fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f657415dd70 0x7f657409a570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56304/0 (socket says 172.21.15.12:56304) 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6572ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f657415e150 0x7f657406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51714/0 (socket says 172.21.15.12:51714) 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f65737fe640 1 -- 172.21.15.12:0/1875421497 learned_addr learned my addr 172.21.15.12:0/1875421497 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6573fff640 1 --2- 172.21.15.12:0/1875421497 >> v2:172.21.15.156:3300/0 conn(0x7f657406cb10 0x7f65740cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6572ffd640 1 -- 172.21.15.12:0/1875421497 >> v2:172.21.15.156:3300/0 conn(0x7f657406cb10 msgr2=0x7f65740cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6572ffd640 1 --2- 172.21.15.12:0/1875421497 >> v2:172.21.15.156:3300/0 conn(0x7f657406cb10 0x7f65740cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6572ffd640 1 -- 172.21.15.12:0/1875421497 >> v1:172.21.15.12:6789/0 conn(0x7f657415dd70 legacy=0x7f657409a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f6572ffd640 1 -- 172.21.15.12:0/1875421497 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657415ed70 con 0x7f657415e150 2024-03-08T08:29:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.334+0000 7f65727fc640 1 -- 172.21.15.12:0/1875421497 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593076656 0 0) 0x7f657415e7b0 con 0x7f657415dd70 2024-03-08T08:29:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f6572ffd640 1 --2- 172.21.15.12:0/1875421497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657406c5d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f657406c560 tx=0x7f656802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3726d96cd43c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f65727fc640 1 -- 172.21.15.12:0/1875421497 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680315a0 con 0x7f657415e150 2024-03-08T08:29:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f65727fc640 1 -- 172.21.15.12:0/1875421497 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568037070 con 0x7f657415e150 2024-03-08T08:29:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f65727fc640 1 -- 172.21.15.12:0/1875421497 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568031960 con 0x7f657415e150 2024-03-08T08:29:49.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 -- 172.21.15.12:0/1875421497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 msgr2=0x7f657406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1875421497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657406c5d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f657406c560 tx=0x7f656802b840 comp rx=0 tx=0).stop 2024-03-08T08:29:49.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 -- 172.21.15.12:0/1875421497 shutdown_connections 2024-03-08T08:29:49.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1875421497 >> v2:172.21.15.156:3300/0 conn(0x7f657406cb10 0x7f65740cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1875421497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657406c5d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 -- 172.21.15.12:0/1875421497 >> 172.21.15.12:0/1875421497 conn(0x7f65740b3370 msgr2=0x7f65740b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 -- 172.21.15.12:0/1875421497 shutdown_connections 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.335+0000 7f657a1c7640 1 -- 172.21.15.12:0/1875421497 wait complete. 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.336+0000 7f657a1c7640 1 Processor -- start 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 -- start start 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 0x7f6574151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f657415dd70 0x7f657409c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740d0190 con 0x7f657415e150 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657406c140 con 0x7f657406cb10 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f657a1c7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740d0310 con 0x7f657415dd70 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f6573fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f6573fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51724/0 (socket says 172.21.15.12:51724) 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f6573fff640 1 -- 172.21.15.12:0/1962160182 learned_addr learned my addr 172.21.15.12:0/1962160182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.337+0000 7f6572ffd640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f657415dd70 0x7f657409c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f65737fe640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 0x7f6574151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 -- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f657415dd70 msgr2=0x7f657409c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f657415dd70 0x7f657409c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 -- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 msgr2=0x7f6574151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 0x7f6574151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 -- 172.21.15.12:0/1962160182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657409f170 con 0x7f657415e150 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f65737fe640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 0x7f6574151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f6573fff640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f656c007e40 tx=0x7f656c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c014620 con 0x7f657415e150 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c019070 con 0x7f657415e150 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c014a20 con 0x7f657415e150 2024-03-08T08:29:49.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657409d8d0 con 0x7f657415e150 2024-03-08T08:29:49.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.338+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657409bb70 con 0x7f657415e150 2024-03-08T08:29:49.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.339+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65740749c0 con 0x7f657415e150 2024-03-08T08:29:49.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.339+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f656c014bc0 con 0x7f657415e150 2024-03-08T08:29:49.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.340+0000 7f655bfff640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.340+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f656c08aad0 con 0x7f657415e150 2024-03-08T08:29:49.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.340+0000 7f65737fe640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.340+0000 7f65737fe640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6564000f60 tx=0x7f6564009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.344+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f656c059950 con 0x7f657415e150 2024-03-08T08:29:49.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.522+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f657406d230 con 0x7f657415e150 2024-03-08T08:29:49.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.523+0000 7f655bfff640 1 -- 172.21.15.12:0/1962160182 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f656c059250 con 0x7f657415e150 2024-03-08T08:29:49.524 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:49.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 msgr2=0x7f656006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6564000f60 tx=0x7f6564009040 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 msgr2=0x7f657409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f656c007e40 tx=0x7f656c00e4b0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 shutdown_connections 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657415e150 0x7f657409cd60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f657415dd70 0x7f657409c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 --2- 172.21.15.12:0/1962160182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f657406cb10 0x7f6574151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 >> 172.21.15.12:0/1962160182 conn(0x7f65740b3370 msgr2=0x7f65740af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 shutdown_connections 2024-03-08T08:29:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.525+0000 7f657a1c7640 1 -- 172.21.15.12:0/1962160182 wait complete. 2024-03-08T08:29:49.537 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.639+0000 7f534c15e640 1 Processor -- start 2024-03-08T08:29:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.639+0000 7f534c15e640 1 -- start start 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534c15e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f53440cc390 0x7f53440cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534c15e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5344099b90 0x7f534409a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534c15e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440cdf60 con 0x7f534415bb90 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534c15e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440cd590 con 0x7f53440cc390 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534c15e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440cf700 con 0x7f5344099b90 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f53496f9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f53440cc390 0x7f53440cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f53496f9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f53440cc390 0x7f53440cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51738/0 (socket says 172.21.15.12:51738) 2024-03-08T08:29:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f53496f9640 1 -- 172.21.15.12:0/4213500105 learned_addr learned my addr 172.21.15.12:0/4213500105 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.640+0000 7f534a6fb640 1 --2- 172.21.15.12:0/4213500105 >> v2:172.21.15.156:3300/0 conn(0x7f5344099b90 0x7f534409a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f53496f9640 1 -- 172.21.15.12:0/4213500105 >> v2:172.21.15.156:3300/0 conn(0x7f5344099b90 msgr2=0x7f534409a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f53496f9640 1 --2- 172.21.15.12:0/4213500105 >> v2:172.21.15.156:3300/0 conn(0x7f5344099b90 0x7f534409a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f53496f9640 1 -- 172.21.15.12:0/4213500105 >> v1:172.21.15.156:6789/0 conn(0x7f534415bb90 legacy=0x7f53440d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f53496f9640 1 -- 172.21.15.12:0/4213500105 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440cc980 con 0x7f53440cc390 2024-03-08T08:29:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f53496f9640 1 --2- 172.21.15.12:0/4213500105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53440cc390 0x7f53440cda20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f534409ac80 tx=0x7f533802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a12d895ae47b9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f5348ef8640 1 -- 172.21.15.12:0/4213500105 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338035020 con 0x7f53440cc390 2024-03-08T08:29:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f5348ef8640 1 -- 172.21.15.12:0/4213500105 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338031870 con 0x7f53440cc390 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.641+0000 7f5348ef8640 1 -- 172.21.15.12:0/4213500105 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338031c40 con 0x7f53440cc390 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 -- 172.21.15.12:0/4213500105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53440cc390 msgr2=0x7f53440cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 --2- 172.21.15.12:0/4213500105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53440cc390 0x7f53440cda20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f534409ac80 tx=0x7f533802b910 comp rx=0 tx=0).stop 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 -- 172.21.15.12:0/4213500105 shutdown_connections 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 --2- 172.21.15.12:0/4213500105 >> v2:172.21.15.156:3300/0 conn(0x7f5344099b90 0x7f534409a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 --2- 172.21.15.12:0/4213500105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53440cc390 0x7f53440cda20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 -- 172.21.15.12:0/4213500105 >> 172.21.15.12:0/4213500105 conn(0x7f53440b3370 msgr2=0x7f53440b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 -- 172.21.15.12:0/4213500105 shutdown_connections 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.642+0000 7f534c15e640 1 -- 172.21.15.12:0/4213500105 wait complete. 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.643+0000 7f534c15e640 1 Processor -- start 2024-03-08T08:29:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.643+0000 7f534c15e640 1 -- start start 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 0x7f5344152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 0x7f53440b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440cd220 con 0x7f534415bb90 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440ccf40 con 0x7f5344099b90 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534c15e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534415bf70 con 0x7f53440cc390 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51750/0 (socket says 172.21.15.12:51750) 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 -- 172.21.15.12:0/2689635529 learned_addr learned my addr 172.21.15.12:0/2689635529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f53496f9640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 0x7f53440b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f5349efa640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 0x7f5344152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 -- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 msgr2=0x7f53440b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 0x7f53440b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 -- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 msgr2=0x7f5344152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 0x7f5344152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f534a6fb640 1 -- 172.21.15.12:0/2689635529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440b5000 con 0x7f534415bb90 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f5349efa640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 0x7f5344152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.644+0000 7f53496f9640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 0x7f53440b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f534a6fb640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5334007e70 tx=0x7f533400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334014620 con 0x7f534415bb90 2024-03-08T08:29:49.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5334019070 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53440b52e0 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334014a20 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.645+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53440b58a0 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.646+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440749c0 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.646+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5334014bc0 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.647+0000 7f53427fc640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.647+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f533408ab00 con 0x7f534415bb90 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.647+0000 7f5349efa640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.647+0000 7f5349efa640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f533c006fd0 tx=0x7f533c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.651+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5334059980 con 0x7f534415bb90 2024-03-08T08:29:49.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.830+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53440c4ea0 con 0x7f534415bb90 2024-03-08T08:29:49.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.831+0000 7f53427fc640 1 -- 172.21.15.12:0/2689635529 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5334059280 con 0x7f534415bb90 2024-03-08T08:29:49.832 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:49.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 msgr2=0x7f533006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f533c006fd0 tx=0x7f533c008040 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 msgr2=0x7f53440c5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5334007e70 tx=0x7f533400e4b0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 shutdown_connections 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534415bb90 0x7f53440c5f30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53440cc390 0x7f53440b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 --2- 172.21.15.12:0/2689635529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5344099b90 0x7f5344152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 >> 172.21.15.12:0/2689635529 conn(0x7f53440b3370 msgr2=0x7f53440c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 shutdown_connections 2024-03-08T08:29:49.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.833+0000 7f534c15e640 1 -- 172.21.15.12:0/2689635529 wait complete. 2024-03-08T08:29:49.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.948+0000 7f98bc2eb640 1 Processor -- start 2024-03-08T08:29:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.948+0000 7f98bc2eb640 1 -- start start 2024-03-08T08:29:49.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98bc2eb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98b40d0250 0x7f98b40d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98bc2eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d25d0 0x7f98b40d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98bc2eb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b40b6be0 con 0x7f98b415dec0 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98bc2eb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40d2140 con 0x7f98b40d0250 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98bc2eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415e2a0 con 0x7f98b40d25d0 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98b40d0250 0x7f98b40d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98ba087640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f98b415dec0 0x7f98b4161990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56316/0 (socket says 172.21.15.12:56316) 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98b40d0250 0x7f98b40d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51760/0 (socket says 172.21.15.12:51760) 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98ba087640 1 -- 172.21.15.12:0/3300323938 learned_addr learned my addr 172.21.15.12:0/3300323938 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98ba888640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d25d0 0x7f98b40d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 -- 172.21.15.12:0/3300323938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d25d0 msgr2=0x7f98b40d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d25d0 0x7f98b40d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 -- 172.21.15.12:0/3300323938 >> v1:172.21.15.12:6789/0 conn(0x7f98b415dec0 legacy=0x7f98b4161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9886640 1 -- 172.21.15.12:0/3300323938 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b41620a0 con 0x7f98b40d0250 2024-03-08T08:29:49.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.949+0000 7f98b9085640 1 -- 172.21.15.12:0/3300323938 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044124651 0 0) 0x7f98ac001110 con 0x7f98b415dec0 2024-03-08T08:29:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.950+0000 7f98b9886640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d0250 0x7f98b40d06b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f98a80097a0 tx=0x7f98a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def8fc66dd251ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.950+0000 7f98b9085640 1 -- 172.21.15.12:0/3300323938 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8035020 con 0x7f98b40d0250 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.950+0000 7f98b9085640 1 -- 172.21.15.12:0/3300323938 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a8031830 con 0x7f98b40d0250 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.950+0000 7f98b9085640 1 -- 172.21.15.12:0/3300323938 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8031c00 con 0x7f98b40d0250 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 -- 172.21.15.12:0/3300323938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d0250 msgr2=0x7f98b40d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d0250 0x7f98b40d06b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f98a80097a0 tx=0x7f98a802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 -- 172.21.15.12:0/3300323938 shutdown_connections 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d25d0 0x7f98b40d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/3300323938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d0250 0x7f98b40d06b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 -- 172.21.15.12:0/3300323938 >> 172.21.15.12:0/3300323938 conn(0x7f98b406c240 msgr2=0x7f98b406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 -- 172.21.15.12:0/3300323938 shutdown_connections 2024-03-08T08:29:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.951+0000 7f98bc2eb640 1 -- 172.21.15.12:0/3300323938 wait complete. 2024-03-08T08:29:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.952+0000 7f98bc2eb640 1 Processor -- start 2024-03-08T08:29:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.952+0000 7f98bc2eb640 1 -- start start 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d0250 0x7f98b415f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 0x7f98b40b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4162630 con 0x7f98b40d25d0 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4162350 con 0x7f98b415dec0 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98bc2eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40d1dd0 con 0x7f98b40d0250 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51774/0 (socket says 172.21.15.12:51774) 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 -- 172.21.15.12:0/2595904701 learned_addr learned my addr 172.21.15.12:0/2595904701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98ba888640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 0x7f98b40b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 -- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d0250 msgr2=0x7f98b415f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d0250 0x7f98b415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 -- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 msgr2=0x7f98b40b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 0x7f98b40b46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 -- 172.21.15.12:0/2595904701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b415e550 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98ba888640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 0x7f98b40b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.953+0000 7f98b9886640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f98a802bee0 tx=0x7f98a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.954+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8041070 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.954+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a803a950 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.954+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b415e830 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.954+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8035040 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.954+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b415edf0 con 0x7f98b40d25d0 2024-03-08T08:29:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.955+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b40749c0 con 0x7f98b40d25d0 2024-03-08T08:29:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.955+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98a8038020 con 0x7f98b40d25d0 2024-03-08T08:29:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.955+0000 7f98b2ffd640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.956+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a80abbf0 con 0x7f98b40d25d0 2024-03-08T08:29:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.956+0000 7f98ba087640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.956+0000 7f98ba087640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 0x7f98a006d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005880 tx=0x7f98ac007a00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:49.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:49.960+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f98a807ab00 con 0x7f98b40d25d0 2024-03-08T08:29:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.139+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98b415efc0 con 0x7f98b40d25d0 2024-03-08T08:29:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.140+0000 7f98b2ffd640 1 -- 172.21.15.12:0/2595904701 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98a807a400 con 0x7f98b40d25d0 2024-03-08T08:29:50.141 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 msgr2=0x7f98a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 0x7f98a006d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005880 tx=0x7f98ac007a00 comp rx=0 tx=0).stop 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 msgr2=0x7f98b414edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f98a802bee0 tx=0x7f98a80046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 shutdown_connections 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b415dec0 0x7f98b40b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98b40d25d0 0x7f98b414edd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 --2- 172.21.15.12:0/2595904701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98b40d0250 0x7f98b415f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 >> 172.21.15.12:0/2595904701 conn(0x7f98b406c240 msgr2=0x7f98b40b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 shutdown_connections 2024-03-08T08:29:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.142+0000 7f98bc2eb640 1 -- 172.21.15.12:0/2595904701 wait complete. 2024-03-08T08:29:50.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.255+0000 7fe72c6f4640 1 Processor -- start 2024-03-08T08:29:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.255+0000 7fe72c6f4640 1 -- start start 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 0x7fe72409a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe72409ae10 0x7fe724099b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7240cd4f0 0x7fe72409a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7240cc730 con 0x7fe7240cd4f0 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7240cca10 con 0x7fe72415bbe0 2024-03-08T08:29:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72c6f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240cccf0 con 0x7fe72409ae10 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72ac91640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7240cd4f0 0x7fe72409a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72ac91640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7240cd4f0 0x7fe72409a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51784/0 (socket says 172.21.15.12:51784) 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.256+0000 7fe72ac91640 1 -- 172.21.15.12:0/2276089065 learned_addr learned my addr 172.21.15.12:0/2276089065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72a490640 1 --2- 172.21.15.12:0/2276089065 >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 0x7fe72409a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe729c8f640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe72409ae10 0x7fe724099b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72ac91640 1 -- 172.21.15.12:0/2276089065 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe72409ae10 msgr2=0x7fe724099b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72ac91640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe72409ae10 0x7fe724099b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72ac91640 1 -- 172.21.15.12:0/2276089065 >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 msgr2=0x7fe72409a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72ac91640 1 --2- 172.21.15.12:0/2276089065 >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 0x7fe72409a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72ac91640 1 -- 172.21.15.12:0/2276089065 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240ccfd0 con 0x7fe7240cd4f0 2024-03-08T08:29:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.257+0000 7fe72a490640 1 --2- 172.21.15.12:0/2276089065 >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 0x7fe72409a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72ac91640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240cd4f0 0x7fe72409a0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe724073300 tx=0x7fe71c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b354e515c7c73f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72948e640 1 -- 172.21.15.12:0/2276089065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c02ca70 con 0x7fe7240cd4f0 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72948e640 1 -- 172.21.15.12:0/2276089065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c02cc10 con 0x7fe7240cd4f0 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72948e640 1 -- 172.21.15.12:0/2276089065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0343f0 con 0x7fe7240cd4f0 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 -- 172.21.15.12:0/2276089065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240cd4f0 msgr2=0x7fe72409a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240cd4f0 0x7fe72409a0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe724073300 tx=0x7fe71c02bed0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 -- 172.21.15.12:0/2276089065 shutdown_connections 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240cd4f0 0x7fe72409a0d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/2276089065 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe72409ae10 0x7fe724099b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/2276089065 >> v2:172.21.15.156:3300/0 conn(0x7fe72415bbe0 0x7fe72409a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 -- 172.21.15.12:0/2276089065 >> 172.21.15.12:0/2276089065 conn(0x7fe7240b3370 msgr2=0x7fe7240b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 -- 172.21.15.12:0/2276089065 shutdown_connections 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.258+0000 7fe72c6f4640 1 -- 172.21.15.12:0/2276089065 wait complete. 2024-03-08T08:29:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.259+0000 7fe72c6f4640 1 Processor -- start 2024-03-08T08:29:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 -- start start 2024-03-08T08:29:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe72409ae10 0x7fe7241f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7240cd4f0 0x7fe7241f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72415c140 con 0x7fe72415bbe0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72415bfc0 con 0x7fe72409ae10 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72c6f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240d0690 con 0x7fe7240cd4f0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72ac91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72ac91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51790/0 (socket says 172.21.15.12:51790) 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.260+0000 7fe72ac91640 1 -- 172.21.15.12:0/79673704 learned_addr learned my addr 172.21.15.12:0/79673704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 -- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7240cd4f0 msgr2=0x7fe7241f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72a490640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe72409ae10 0x7fe7241f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe729c8f640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7240cd4f0 0x7fe7241f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7240cd4f0 0x7fe7241f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 -- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe72409ae10 msgr2=0x7fe7241f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe72409ae10 0x7fe7241f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 -- 172.21.15.12:0/79673704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7241f7de0 con 0x7fe72415bbe0 2024-03-08T08:29:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72ac91640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009f40 tx=0x7fe71c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c043020 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c03e450 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72406d930 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c009510 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.261+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7240b5720 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.262+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7240749c0 con 0x7fe72415bbe0 2024-03-08T08:29:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.262+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe71c046020 con 0x7fe72415bbe0 2024-03-08T08:29:50.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.263+0000 7fe71affd640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.263+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe71c0acba0 con 0x7fe72415bbe0 2024-03-08T08:29:50.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.263+0000 7fe72a490640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.263+0000 7fe72a490640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe720006fd0 tx=0x7fe720008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.267+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe71c04d030 con 0x7fe72415bbe0 2024-03-08T08:29:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.459+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe7240b4260 con 0x7fe72415bbe0 2024-03-08T08:29:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.459+0000 7fe71affd640 1 -- 172.21.15.12:0/79673704 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe71c07ba20 con 0x7fe72415bbe0 2024-03-08T08:29:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 msgr2=0x7fe70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe720006fd0 tx=0x7fe720008040 comp rx=0 tx=0).stop 2024-03-08T08:29:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 msgr2=0x7fe7240b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009f40 tx=0x7fe71c004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 shutdown_connections 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe72415bbe0 0x7fe7240b6200 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7240cd4f0 0x7fe7241f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 --2- 172.21.15.12:0/79673704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe72409ae10 0x7fe7241f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 >> 172.21.15.12:0/79673704 conn(0x7fe7240b3370 msgr2=0x7fe7240c5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 shutdown_connections 2024-03-08T08:29:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.461+0000 7fe72c6f4640 1 -- 172.21.15.12:0/79673704 wait complete. 2024-03-08T08:29:50.473 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:29:50.473 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:29:50.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:50.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.593+0000 7f7f1982e640 1 Processor -- start 2024-03-08T08:29:50.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.593+0000 7f7f1982e640 1 -- start start 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f1982e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1415ddc0 0x7f7f140d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f1982e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1415e1a0 con 0x7f7f140b7bf0 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f1982e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b8790 con 0x7f7f140d0f40 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f1982e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140b9030 con 0x7f7f1415ddc0 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1415ddc0 0x7f7f140d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1415ddc0 0x7f7f140d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51794/0 (socket says 172.21.15.12:51794) 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 -- 172.21.15.12:0/3966505833 learned_addr learned my addr 172.21.15.12:0/3966505833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f137fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7f140b7bf0 0x7f7f140d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56332/0 (socket says 172.21.15.12:56332) 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 -- 172.21.15.12:0/3966505833 >> v1:172.21.15.156:6789/0 conn(0x7f7f140d0f40 legacy=0x7f7f140d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 -- 172.21.15.12:0/3966505833 >> v1:172.21.15.12:6789/0 conn(0x7f7f140b7bf0 legacy=0x7f7f140d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f11ffb640 1 -- 172.21.15.12:0/3966505833 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562524134 0 0) 0x7f7f1415e1a0 con 0x7f7f140b7bf0 2024-03-08T08:29:50.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.594+0000 7f7f12ffd640 1 -- 172.21.15.12:0/3966505833 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140b8a70 con 0x7f7f1415ddc0 2024-03-08T08:29:50.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.595+0000 7f7f12ffd640 1 --2- 172.21.15.12:0/3966505833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140d0a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009e50 tx=0x7f7f0402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c6c42a8fd9da6d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.595+0000 7f7f11ffb640 1 -- 172.21.15.12:0/3966505833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0402cb20 con 0x7f7f1415ddc0 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.595+0000 7f7f11ffb640 1 -- 172.21.15.12:0/3966505833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0402ccc0 con 0x7f7f1415ddc0 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.595+0000 7f7f11ffb640 1 -- 172.21.15.12:0/3966505833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04034610 con 0x7f7f1415ddc0 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 -- 172.21.15.12:0/3966505833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 msgr2=0x7f7f140d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 --2- 172.21.15.12:0/3966505833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140d0a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009e50 tx=0x7f7f0402bf80 comp rx=0 tx=0).stop 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 -- 172.21.15.12:0/3966505833 shutdown_connections 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 --2- 172.21.15.12:0/3966505833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140d0a00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 -- 172.21.15.12:0/3966505833 >> 172.21.15.12:0/3966505833 conn(0x7f7f14072720 msgr2=0x7f7f14072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 -- 172.21.15.12:0/3966505833 shutdown_connections 2024-03-08T08:29:50.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.596+0000 7f7f1982e640 1 -- 172.21.15.12:0/3966505833 wait complete. 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.597+0000 7f7f1982e640 1 Processor -- start 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.597+0000 7f7f1982e640 1 -- start start 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f140b7bf0 0x7f7f14157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f140d0f40 0x7f7f141f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140b9980 con 0x7f7f1415ddc0 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140d1d20 con 0x7f7f140b7bf0 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f1982e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140d2fd0 con 0x7f7f140d0f40 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51796/0 (socket says 172.21.15.12:51796) 2024-03-08T08:29:50.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 -- 172.21.15.12:0/4222436955 learned_addr learned my addr 172.21.15.12:0/4222436955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 -- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f140d0f40 msgr2=0x7f7f141f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f127fc640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f140d0f40 0x7f7f141f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f12ffd640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f140b7bf0 0x7f7f14157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f140d0f40 0x7f7f141f36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 -- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f140b7bf0 msgr2=0x7f7f14157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f140b7bf0 0x7f7f14157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 -- 172.21.15.12:0/4222436955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14157750 con 0x7f7f1415ddc0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.598+0000 7f7f137fe640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00c450 tx=0x7f7f0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.599+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c018070 con 0x7f7f1415ddc0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.599+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c007500 con 0x7f7f1415ddc0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.599+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f141f4140 con 0x7f7f1415ddc0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.599+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c004a20 con 0x7f7f1415ddc0 2024-03-08T08:29:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.599+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1415fbf0 con 0x7f7f1415ddc0 2024-03-08T08:29:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.600+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f140749c0 con 0x7f7f1415ddc0 2024-03-08T08:29:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.600+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f0c01b070 con 0x7f7f1415ddc0 2024-03-08T08:29:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.600+0000 7f7f0bfff640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 0x7f7ef406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.601+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f0c089380 con 0x7f7f1415ddc0 2024-03-08T08:29:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.601+0000 7f7f12ffd640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 0x7f7ef406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.601+0000 7f7f12ffd640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 0x7f7ef406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7f04035000 tx=0x7f7f04030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.605+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7f14157750 con 0x7f7f1415ddc0 2024-03-08T08:29:50.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.785+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f140d2500 con 0x7f7f1415ddc0 2024-03-08T08:29:50.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.786+0000 7f7f0bfff640 1 -- 172.21.15.12:0/4222436955 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f7f0c058290 con 0x7f7f1415ddc0 2024-03-08T08:29:50.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:50.788 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:50.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 msgr2=0x7f7ef406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 0x7f7ef406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7f04035000 tx=0x7f7f04030000 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 msgr2=0x7f7f140b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00c450 tx=0x7f7f0c00c910 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 shutdown_connections 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7ef406b4d0 0x7f7ef406d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1415ddc0 0x7f7f140b3e40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f140d0f40 0x7f7f141f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 --2- 172.21.15.12:0/4222436955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f140b7bf0 0x7f7f14157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 >> 172.21.15.12:0/4222436955 conn(0x7f7f14072720 msgr2=0x7f7f140b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 shutdown_connections 2024-03-08T08:29:50.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.788+0000 7f7f1982e640 1 -- 172.21.15.12:0/4222436955 wait complete. 2024-03-08T08:29:50.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:50.800 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:50.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.903+0000 7f678fc9a640 1 Processor -- start 2024-03-08T08:29:50.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.903+0000 7f678fc9a640 1 -- start start 2024-03-08T08:29:50.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678fc9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67880b9760 0x7f67880d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678fc9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880d2050 0x7f67880b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678fc9a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880b7250 con 0x7f678815dbc0 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678fc9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880b7520 con 0x7f67880b9760 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678fc9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880b7800 con 0x7f67880d2050 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678d235640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67880b9760 0x7f67880d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678d235640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67880b9760 0x7f67880d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51800/0 (socket says 172.21.15.12:51800) 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678d235640 1 -- 172.21.15.12:0/2350363344 learned_addr learned my addr 172.21.15.12:0/2350363344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.904+0000 7f678da36640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f678815dbc0 0x7f67880b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56334/0 (socket says 172.21.15.12:56334) 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678d235640 1 -- 172.21.15.12:0/2350363344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880d2050 msgr2=0x7f67880b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678d235640 1 --2- 172.21.15.12:0/2350363344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880d2050 0x7f67880b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678d235640 1 -- 172.21.15.12:0/2350363344 >> v1:172.21.15.12:6789/0 conn(0x7f678815dbc0 legacy=0x7f67880b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678d235640 1 -- 172.21.15.12:0/2350363344 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67880b7ae0 con 0x7f67880b9760 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678d235640 1 --2- 172.21.15.12:0/2350363344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67880b9760 0x7f67880d1b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f677c009870 tx=0x7f677c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad8390647afe11a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.905+0000 7f678ca34640 1 -- 172.21.15.12:0/2350363344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c02f830 con 0x7f67880b9760 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678ca34640 1 -- 172.21.15.12:0/2350363344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c02f9d0 con 0x7f67880b9760 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678ca34640 1 -- 172.21.15.12:0/2350363344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c02fda0 con 0x7f67880b9760 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 -- 172.21.15.12:0/2350363344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67880b9760 msgr2=0x7f67880d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 --2- 172.21.15.12:0/2350363344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67880b9760 0x7f67880d1b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f677c009870 tx=0x7f677c02ba10 comp rx=0 tx=0).stop 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 -- 172.21.15.12:0/2350363344 shutdown_connections 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 --2- 172.21.15.12:0/2350363344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880d2050 0x7f67880b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 --2- 172.21.15.12:0/2350363344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67880b9760 0x7f67880d1b10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 -- 172.21.15.12:0/2350363344 >> 172.21.15.12:0/2350363344 conn(0x7f6788072720 msgr2=0x7f6788072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 -- 172.21.15.12:0/2350363344 shutdown_connections 2024-03-08T08:29:50.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.906+0000 7f678fc9a640 1 -- 172.21.15.12:0/2350363344 wait complete. 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.907+0000 7f678fc9a640 1 Processor -- start 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.907+0000 7f678fc9a640 1 -- start start 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 0x7f678814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 0x7f67881e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678815dfa0 con 0x7f678815dbc0 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880b80a0 con 0x7f67880d2050 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678fc9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880b8b00 con 0x7f67880b9760 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51810/0 (socket says 172.21.15.12:51810) 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 -- 172.21.15.12:0/1088467502 learned_addr learned my addr 172.21.15.12:0/1088467502 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678d235640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 0x7f67881e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678da36640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 0x7f678814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 -- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 msgr2=0x7f678814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 0x7f678814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 -- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 msgr2=0x7f67881e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 0x7f67881e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.908+0000 7f678e237640 1 -- 172.21.15.12:0/1088467502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67881f7d70 con 0x7f678815dbc0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f678d235640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 0x7f67881e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f678da36640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 0x7f678814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f678e237640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f678000b420 tx=0x7f678000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780011830 con 0x7f678815dbc0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780016070 con 0x7f678815dbc0 2024-03-08T08:29:50.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780011c10 con 0x7f678815dbc0 2024-03-08T08:29:50.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67880cbaf0 con 0x7f678815dbc0 2024-03-08T08:29:50.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.909+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67880cc0b0 con 0x7f678815dbc0 2024-03-08T08:29:50.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.910+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67880b6ff0 con 0x7f678815dbc0 2024-03-08T08:29:50.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.910+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f678001b460 con 0x7f678815dbc0 2024-03-08T08:29:50.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.911+0000 7f67867fc640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 0x7f677406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:50.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.911+0000 7f678da36640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 0x7f677406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:50.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.911+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6780089eb0 con 0x7f678815dbc0 2024-03-08T08:29:50.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.911+0000 7f678da36640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 0x7f677406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67881f6ef0 tx=0x7f6778009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:50.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:50.915+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f67880b6ff0 con 0x7f678815dbc0 2024-03-08T08:29:51.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.094+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67880b5f60 con 0x7f678815dbc0 2024-03-08T08:29:51.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.094+0000 7f67867fc640 1 -- 172.21.15.12:0/1088467502 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6780058d30 con 0x7f678815dbc0 2024-03-08T08:29:51.095 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 msgr2=0x7f677406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 0x7f677406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67881f6ef0 tx=0x7f6778009040 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 msgr2=0x7f67881f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f678000b420 tx=0x7f678000b8e0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 shutdown_connections 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f677406b500 0x7f677406d9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f678815dbc0 0x7f67881f6f90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67880d2050 0x7f67881e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 --2- 172.21.15.12:0/1088467502 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67880b9760 0x7f678814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 >> 172.21.15.12:0/1088467502 conn(0x7f6788072720 msgr2=0x7f67880b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 shutdown_connections 2024-03-08T08:29:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.096+0000 7f678fc9a640 1 -- 172.21.15.12:0/1088467502 wait complete. 2024-03-08T08:29:51.108 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:51.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.211+0000 7f7e49b3e640 1 Processor -- start 2024-03-08T08:29:51.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.211+0000 7f7e49b3e640 1 -- start start 2024-03-08T08:29:51.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e49b3e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e44161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e49b3e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 0x7f7e440d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e49b3e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440b6be0 con 0x7f7e44161ed0 2024-03-08T08:29:51.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e49b3e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e440d2140 con 0x7f7e440d0960 2024-03-08T08:29:51.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e49b3e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4415e2a0 con 0x7f7e4415dec0 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e43fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 0x7f7e440d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e437fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e44161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e42ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7e44161ed0 0x7f7e440d0250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:52656/0 (socket says 172.21.15.12:52656) 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e42ffd640 1 -- 172.21.15.12:0/736912698 learned_addr learned my addr 172.21.15.12:0/736912698 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.212+0000 7f7e437fe640 1 -- 172.21.15.12:0/736912698 >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 msgr2=0x7f7e440d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e437fe640 1 --2- 172.21.15.12:0/736912698 >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 0x7f7e440d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e437fe640 1 -- 172.21.15.12:0/736912698 >> v1:172.21.15.156:6789/0 conn(0x7f7e44161ed0 legacy=0x7f7e440d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e437fe640 1 -- 172.21.15.12:0/736912698 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e44162350 con 0x7f7e4415dec0 2024-03-08T08:29:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e43fff640 1 --2- 172.21.15.12:0/736912698 >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 0x7f7e440d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e437fe640 1 --2- 172.21.15.12:0/736912698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e44161990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e340097a0 tx=0x7f7e3402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42adf6244b3edd66 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.213+0000 7f7e427fc640 1 -- 172.21.15.12:0/736912698 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34035020 con 0x7f7e4415dec0 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e427fc640 1 -- 172.21.15.12:0/736912698 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e34031830 con 0x7f7e4415dec0 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e427fc640 1 -- 172.21.15.12:0/736912698 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34031c00 con 0x7f7e4415dec0 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 -- 172.21.15.12:0/736912698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 msgr2=0x7f7e44161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/736912698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e44161990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e340097a0 tx=0x7f7e3402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 -- 172.21.15.12:0/736912698 shutdown_connections 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/736912698 >> v2:172.21.15.156:3300/0 conn(0x7f7e440d0960 0x7f7e440d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/736912698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e44161990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 -- 172.21.15.12:0/736912698 >> 172.21.15.12:0/736912698 conn(0x7f7e4406c240 msgr2=0x7f7e4406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 -- 172.21.15.12:0/736912698 shutdown_connections 2024-03-08T08:29:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.214+0000 7f7e49b3e640 1 -- 172.21.15.12:0/736912698 wait complete. 2024-03-08T08:29:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.215+0000 7f7e49b3e640 1 Processor -- start 2024-03-08T08:29:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.215+0000 7f7e49b3e640 1 -- start start 2024-03-08T08:29:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440d0960 0x7f7e441590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e441f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440b94f0 con 0x7f7e44161ed0 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440d1dd0 con 0x7f7e440d0960 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e49b3e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440b9670 con 0x7f7e4415dec0 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51820/0 (socket says 172.21.15.12:51820) 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e43fff640 1 -- 172.21.15.12:0/3964890246 learned_addr learned my addr 172.21.15.12:0/3964890246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e437fe640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440d0960 0x7f7e441590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.216+0000 7f7e42ffd640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e441f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 -- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 msgr2=0x7f7e441f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e441f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 -- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440d0960 msgr2=0x7f7e441590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440d0960 0x7f7e441590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 -- 172.21.15.12:0/3964890246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e441595e0 con 0x7f7e44161ed0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e43fff640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009db0 tx=0x7f7e3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c018030 con 0x7f7e44161ed0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00cd30 con 0x7f7e44161ed0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e441f4520 con 0x7f7e44161ed0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c003bb0 con 0x7f7e44161ed0 2024-03-08T08:29:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.217+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e441f4ae0 con 0x7f7e44161ed0 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.218+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e440b2ba0 con 0x7f7e44161ed0 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.219+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e3c005d10 con 0x7f7e44161ed0 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.219+0000 7f7e2bfff640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.219+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e3c08a970 con 0x7f7e44161ed0 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.219+0000 7f7e437fe640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.220+0000 7f7e437fe640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 0x7f7e3006d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e34036000 tx=0x7f7e34008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.223+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e440b2ba0 con 0x7f7e44161ed0 2024-03-08T08:29:51.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.405+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e440d1750 con 0x7f7e44161ed0 2024-03-08T08:29:51.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.405+0000 7f7e2bfff640 1 -- 172.21.15.12:0/3964890246 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e3c022070 con 0x7f7e44161ed0 2024-03-08T08:29:51.407 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 msgr2=0x7f7e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 0x7f7e3006d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e34036000 tx=0x7f7e34008030 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 msgr2=0x7f7e44160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009db0 tx=0x7f7e3c00c490 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 shutdown_connections 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e44161ed0 0x7f7e44160ad0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4415dec0 0x7f7e441f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 --2- 172.21.15.12:0/3964890246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440d0960 0x7f7e441590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 >> 172.21.15.12:0/3964890246 conn(0x7f7e4406c240 msgr2=0x7f7e4415be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 shutdown_connections 2024-03-08T08:29:51.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.407+0000 7f7e49b3e640 1 -- 172.21.15.12:0/3964890246 wait complete. 2024-03-08T08:29:51.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:51.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.522+0000 7fcaa7afb640 1 Processor -- start 2024-03-08T08:29:51.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.522+0000 7fcaa7afb640 1 -- start start 2024-03-08T08:29:51.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa7afb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcaa00cd600 0x7fcaa01656b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa7afb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa00b2ca0 con 0x7fcaa015aa30 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa7afb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa00b2f80 con 0x7fcaa015ae10 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa7afb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa006de10 con 0x7fcaa00cd600 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa6098640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcaa00cd600 0x7fcaa01656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa6098640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcaa00cd600 0x7fcaa01656b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51834/0 (socket says 172.21.15.12:51834) 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.523+0000 7fcaa6098640 1 -- 172.21.15.12:0/2134965585 learned_addr learned my addr 172.21.15.12:0/2134965585 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa6098640 1 -- 172.21.15.12:0/2134965585 >> v1:172.21.15.156:6789/0 conn(0x7fcaa015ae10 legacy=0x7fcaa00ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa6098640 1 -- 172.21.15.12:0/2134965585 >> v1:172.21.15.12:6789/0 conn(0x7fcaa015aa30 legacy=0x7fcaa006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa6098640 1 -- 172.21.15.12:0/2134965585 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa00b3260 con 0x7fcaa00cd600 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa6098640 1 --2- 172.21.15.12:0/2134965585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa01656b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fca98009870 tx=0x7fca9802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ceea4a57b748f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa4895640 1 -- 172.21.15.12:0/2134965585 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98034070 con 0x7fcaa00cd600 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa4895640 1 -- 172.21.15.12:0/2134965585 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca98030860 con 0x7fcaa00cd600 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.524+0000 7fcaa4895640 1 -- 172.21.15.12:0/2134965585 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98030c30 con 0x7fcaa00cd600 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 -- 172.21.15.12:0/2134965585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 msgr2=0x7fcaa01656b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/2134965585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa01656b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fca98009870 tx=0x7fca9802b9a0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 -- 172.21.15.12:0/2134965585 shutdown_connections 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/2134965585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa01656b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 -- 172.21.15.12:0/2134965585 >> 172.21.15.12:0/2134965585 conn(0x7fcaa006abd0 msgr2=0x7fcaa006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 -- 172.21.15.12:0/2134965585 shutdown_connections 2024-03-08T08:29:51.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.525+0000 7fcaa7afb640 1 -- 172.21.15.12:0/2134965585 wait complete. 2024-03-08T08:29:51.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.526+0000 7fcaa7afb640 1 Processor -- start 2024-03-08T08:29:51.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.526+0000 7fcaa7afb640 1 -- start start 2024-03-08T08:29:51.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcaa015aa30 0x7fcaa0156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa015ae10 0x7fcaa01424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa006b730 con 0x7fcaa00cd600 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa00b3820 con 0x7fcaa015ae10 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa7afb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa0165bf0 con 0x7fcaa015aa30 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51848/0 (socket says 172.21.15.12:51848) 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 -- 172.21.15.12:0/3930037112 learned_addr learned my addr 172.21.15.12:0/3930037112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 -- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcaa015aa30 msgr2=0x7fcaa0156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcaa015aa30 0x7fcaa0156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 -- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa015ae10 msgr2=0x7fcaa01424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa015ae10 0x7fcaa01424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 -- 172.21.15.12:0/3930037112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa0142c10 con 0x7fcaa00cd600 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.527+0000 7fcaa5897640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca9000c790 tx=0x7fca9000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.528+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90016020 con 0x7fcaa00cd600 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.528+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca90006d50 con 0x7fcaa00cd600 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.528+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90003bb0 con 0x7fcaa00cd600 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.528+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa0142ef0 con 0x7fcaa00cd600 2024-03-08T08:29:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.528+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa01434b0 con 0x7fcaa00cd600 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.529+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa014d700 con 0x7fcaa00cd600 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.529+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca9000a850 con 0x7fcaa00cd600 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.530+0000 7fca9e7fc640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 0x7fca8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.530+0000 7fcaa5096640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 0x7fca8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.530+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca90089f60 con 0x7fcaa00cd600 2024-03-08T08:29:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.530+0000 7fcaa5096640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 0x7fca8c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca94006fd0 tx=0x7fca94006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.534+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fca90020020 con 0x7fcaa00cd600 2024-03-08T08:29:51.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.714+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcaa00749c0 con 0x7fcaa00cd600 2024-03-08T08:29:51.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.714+0000 7fca9e7fc640 1 -- 172.21.15.12:0/3930037112 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca90019310 con 0x7fcaa00cd600 2024-03-08T08:29:51.715 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 msgr2=0x7fca8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 0x7fca8c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca94006fd0 tx=0x7fca94006bc0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 msgr2=0x7fcaa0141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca9000c790 tx=0x7fca9000cc50 comp rx=0 tx=0).stop 2024-03-08T08:29:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 shutdown_connections 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fca8c06b500 0x7fca8c06d9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa015ae10 0x7fcaa01424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcaa015aa30 0x7fcaa0156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 --2- 172.21.15.12:0/3930037112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcaa00cd600 0x7fcaa0141ea0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 >> 172.21.15.12:0/3930037112 conn(0x7fcaa006abd0 msgr2=0x7fcaa0074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 shutdown_connections 2024-03-08T08:29:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.716+0000 7fcaa7afb640 1 -- 172.21.15.12:0/3930037112 wait complete. 2024-03-08T08:29:51.728 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:29:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.828+0000 7f863173e640 1 Processor -- start 2024-03-08T08:29:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.828+0000 7f863173e640 1 -- start start 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f863173e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f862c15ddc0 0x7f862c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f863173e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 0x7f862c0b8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f863173e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862c15e1a0 con 0x7f862c0d0f40 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f863173e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0b9030 con 0x7f862c15ddc0 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f863173e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0d1d20 con 0x7f862c0b7bf0 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f862affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f862c15ddc0 0x7f862c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f862a7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f862c0d0f40 0x7f862c0d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56356/0 (socket says 172.21.15.12:56356) 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.829+0000 7f862a7fc640 1 -- 172.21.15.12:0/504265723 learned_addr learned my addr 172.21.15.12:0/504265723 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862b7fe640 1 --2- 172.21.15.12:0/504265723 >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 0x7f862c0b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862affd640 1 -- 172.21.15.12:0/504265723 >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 msgr2=0x7f862c0b8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862affd640 1 --2- 172.21.15.12:0/504265723 >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 0x7f862c0b8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862affd640 1 -- 172.21.15.12:0/504265723 >> v1:172.21.15.12:6789/0 conn(0x7f862c0d0f40 legacy=0x7f862c0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862affd640 1 -- 172.21.15.12:0/504265723 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0b8790 con 0x7f862c15ddc0 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862b7fe640 1 --2- 172.21.15.12:0/504265723 >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 0x7f862c0b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.830+0000 7f862affd640 1 --2- 172.21.15.12:0/504265723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c15ddc0 0x7f862c0d0a00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f862c0d0470 tx=0x7f862402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4603c4422aba1fd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f8629ffb640 1 -- 172.21.15.12:0/504265723 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624035020 con 0x7f862c15ddc0 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f8629ffb640 1 -- 172.21.15.12:0/504265723 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624031830 con 0x7f862c15ddc0 2024-03-08T08:29:51.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f8629ffb640 1 -- 172.21.15.12:0/504265723 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624031c00 con 0x7f862c15ddc0 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 -- 172.21.15.12:0/504265723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c15ddc0 msgr2=0x7f862c0d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 --2- 172.21.15.12:0/504265723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c15ddc0 0x7f862c0d0a00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f862c0d0470 tx=0x7f862402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 -- 172.21.15.12:0/504265723 shutdown_connections 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 --2- 172.21.15.12:0/504265723 >> v2:172.21.15.156:3300/0 conn(0x7f862c0b7bf0 0x7f862c0b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 --2- 172.21.15.12:0/504265723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c15ddc0 0x7f862c0d0a00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 -- 172.21.15.12:0/504265723 >> 172.21.15.12:0/504265723 conn(0x7f862c072720 msgr2=0x7f862c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:51.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 -- 172.21.15.12:0/504265723 shutdown_connections 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.831+0000 7f863173e640 1 -- 172.21.15.12:0/504265723 wait complete. 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.832+0000 7f863173e640 1 Processor -- start 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.832+0000 7f863173e640 1 -- start start 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 0x7f862c160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 0x7f862c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b9980 con 0x7f862c0d0f40 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b8d50 con 0x7f862c15ddc0 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f863173e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0d28c0 con 0x7f862c0b7bf0 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f862a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f862a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51864/0 (socket says 172.21.15.12:51864) 2024-03-08T08:29:51.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f862a7fc640 1 -- 172.21.15.12:0/2741170954 learned_addr learned my addr 172.21.15.12:0/2741170954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f862affd640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 0x7f862c160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.833+0000 7f862b7fe640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 0x7f862c153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 -- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 msgr2=0x7f862c160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 0x7f862c160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 -- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 msgr2=0x7f862c153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 0x7f862c153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 -- 172.21.15.12:0/2741170954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c157d20 con 0x7f862c0d0f40 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862affd640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 0x7f862c160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862b7fe640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 0x7f862c153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f862a7fc640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f861400c400 tx=0x7f861400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614017070 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8614005ce0 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c1582e0 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86140072f0 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.834+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c1588a0 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.835+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c1609c0 con 0x7f862c0d0f40 2024-03-08T08:29:51.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.835+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f861401a070 con 0x7f862c0d0f40 2024-03-08T08:29:51.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.836+0000 7f86237fe640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 0x7f860c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:51.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.836+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8614089610 con 0x7f862c0d0f40 2024-03-08T08:29:51.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.836+0000 7f862affd640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 0x7f860c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:51.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.836+0000 7f862affd640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 0x7f860c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8624036000 tx=0x7f862403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:51.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:51.840+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f862c1609c0 con 0x7f862c0d0f40 2024-03-08T08:29:52.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.033+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f862c158aa0 con 0x7f862c0d0f40 2024-03-08T08:29:52.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.033+0000 7f86237fe640 1 -- 172.21.15.12:0/2741170954 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f862c158aa0 con 0x7f862c0d0f40 2024-03-08T08:29:52.034 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 msgr2=0x7f860c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 0x7f860c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8624036000 tx=0x7f862403c040 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 msgr2=0x7f862c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f861400c400 tx=0x7f861400c8c0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 shutdown_connections 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860c06b4d0 0x7f860c06d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862c15ddc0 0x7f862c153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f862c0d0f40 0x7f862c157160 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 --2- 172.21.15.12:0/2741170954 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f862c0b7bf0 0x7f862c160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 >> 172.21.15.12:0/2741170954 conn(0x7f862c072720 msgr2=0x7f862c0ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 shutdown_connections 2024-03-08T08:29:52.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.035+0000 7f863173e640 1 -- 172.21.15.12:0/2741170954 wait complete. 2024-03-08T08:29:52.047 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-08T08:29:52.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:52.047 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-08T08:29:52.079 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:52.079 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:29:52.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:29:52.080 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-08T08:29:52.137 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-08T08:29:52.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:52.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.237+0000 7f9778e76640 1 Processor -- start 2024-03-08T08:29:52.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.237+0000 7f9778e76640 1 -- start start 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9778e76640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f977415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9778e76640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f97740d24b0 0x7f97740d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9778e76640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977415ebc0 con 0x7f977406c240 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9778e76640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97740b6b30 con 0x7f97740d24b0 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9778e76640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977415eea0 con 0x7f97740b9420 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f977259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f977406c240 0x7f977406c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56368/0 (socket says 172.21.15.12:56368) 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f977259c640 1 -- 172.21.15.12:0/1459304155 learned_addr learned my addr 172.21.15.12:0/1459304155 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9772d9d640 1 --2- 172.21.15.12:0/1459304155 >> v2:172.21.15.156:3300/0 conn(0x7f97740d24b0 0x7f97740d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534742826 0 0) 0x7f977415ebc0 con 0x7f977406c240 2024-03-08T08:29:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f9771d9b640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f977415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.238+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9760003520 con 0x7f977406c240 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496737063 0 0) 0x7f9760003520 con 0x7f977406c240 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 msgr2=0x7f977415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f977415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 >> v2:172.21.15.156:3300/0 conn(0x7f97740d24b0 msgr2=0x7f97740d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 --2- 172.21.15.12:0/1459304155 >> v2:172.21.15.156:3300/0 conn(0x7f97740d24b0 0x7f97740d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740d1cb0 con 0x7f977406c240 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f976c001670 con 0x7f977406c240 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 >> v1:172.21.15.12:6789/0 conn(0x7f977406c240 legacy=0x7f977406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f9760004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977415eea0 con 0x7f97600039b0 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976000b9c0 con 0x7f9760004d40 2024-03-08T08:29:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976000bb40 con 0x7f97740b9420 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f9772d9d640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f9771d9b640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f9760004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 -- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 msgr2=0x7f9760004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f9760004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 -- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9760004d40 msgr2=0x7f97600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f977259c640 1 -- 172.21.15.12:0/1459304155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760003520 con 0x7f97600039b0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.239+0000 7f9771d9b640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b9420 0x7f9760004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f977259c640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f976c001050 tx=0x7f9760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfcc9303207c0f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c007d70 con 0x7f97600039b0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c030070 con 0x7f97600039b0 2024-03-08T08:29:52.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f977159a640 1 -- 172.21.15.12:0/1459304155 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c02cd80 con 0x7f97600039b0 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f9778e76640 1 -- 172.21.15.12:0/1459304155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97600039b0 msgr2=0x7f9760003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.240+0000 7f9778e76640 1 --2- 172.21.15.12:0/1459304155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f976c001050 tx=0x7f9760005830 comp rx=0 tx=0).stop 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.241+0000 7f9772d9d640 1 -- 172.21.15.12:0/1459304155 reap_dead start 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.241+0000 7f9778e76640 1 -- 172.21.15.12:0/1459304155 shutdown_connections 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.241+0000 7f9778e76640 1 -- 172.21.15.12:0/1459304155 >> 172.21.15.12:0/1459304155 conn(0x7f9774072900 msgr2=0x7f97740d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.241+0000 7f9778e76640 1 -- 172.21.15.12:0/1459304155 shutdown_connections 2024-03-08T08:29:52.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.241+0000 7f9778e76640 1 -- 172.21.15.12:0/1459304155 wait complete. 2024-03-08T08:29:52.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.242+0000 7f9778e76640 1 Processor -- start 2024-03-08T08:29:52.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.242+0000 7f9778e76640 1 -- start start 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97600039b0 0x7f9774155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b56e0 0x7f97740b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977415f180 con 0x7f9774156c40 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977415bc30 con 0x7f97600039b0 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977415e220 con 0x7f97740b56e0 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51880/0 (socket says 172.21.15.12:51880) 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 -- 172.21.15.12:0/1424366821 learned_addr learned my addr 172.21.15.12:0/1424366821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9771d9b640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b56e0 0x7f97740b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f977259c640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97600039b0 0x7f9774155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 -- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b56e0 msgr2=0x7f97740b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b56e0 0x7f97740b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 -- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97600039b0 msgr2=0x7f9774155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97600039b0 0x7f9774155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 -- 172.21.15.12:0/1424366821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97741579b0 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9772d9d640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f977406d280 tx=0x7f976400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976400fde0 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976401d070 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764015670 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.243+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9774157c90 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.244+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97740cab10 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.245+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97740749c0 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.245+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9764015810 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.245+0000 7f976affd640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 0x7f975406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.245+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f976408a990 con 0x7f9774156c40 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.245+0000 7f977259c640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 0x7f975406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.246+0000 7f977259c640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 0x7f975406d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f976c034000 tx=0x7f976c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.250+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9764010040 con 0x7f9774156c40 2024-03-08T08:29:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.465+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97740b3210 con 0x7f9774156c40 2024-03-08T08:29:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.466+0000 7f976affd640 1 -- 172.21.15.12:0/1424366821 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f97640591c0 con 0x7f9774156c40 2024-03-08T08:29:52.467 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:52.467 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 msgr2=0x7f975406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 0x7f975406d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f976c034000 tx=0x7f976c007000 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 msgr2=0x7f97741570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f977406d280 tx=0x7f976400f580 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 shutdown_connections 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f975406b500 0x7f975406d9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9774156c40 0x7f97741570f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f97740b56e0 0x7f97740b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 --2- 172.21.15.12:0/1424366821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97600039b0 0x7f9774155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.467+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 >> 172.21.15.12:0/1424366821 conn(0x7f9774072900 msgr2=0x7f97740b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.468+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 shutdown_connections 2024-03-08T08:29:52.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.468+0000 7f9778e76640 1 -- 172.21.15.12:0/1424366821 wait complete. 2024-03-08T08:29:52.480 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.579+0000 7f4ce918c640 1 Processor -- start 2024-03-08T08:29:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.579+0000 7f4ce918c640 1 -- start start 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce918c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 0x7f4ce40cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce918c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 0x7f4ce40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce918c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce409a960 con 0x7f4ce40d00b0 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce918c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce40d0730 con 0x7f4ce409a580 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce918c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40d0a10 con 0x7f4ce406ce00 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4ce40d00b0 0x7f4ce406c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56380/0 (socket says 172.21.15.12:56380) 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.580+0000 7f4ce259c640 1 -- 172.21.15.12:0/825479862 learned_addr learned my addr 172.21.15.12:0/825479862 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce2d9d640 1 --2- 172.21.15.12:0/825479862 >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 0x7f4ce40cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce359e640 1 --2- 172.21.15.12:0/825479862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 0x7f4ce40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749769649 0 0) 0x7f4ce409a960 con 0x7f4ce40d00b0 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd0003520 con 0x7f4ce40d00b0 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce359e640 1 -- 172.21.15.12:0/825479862 >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 msgr2=0x7f4ce40cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce359e640 1 --2- 172.21.15.12:0/825479862 >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 0x7f4ce40cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce359e640 1 -- 172.21.15.12:0/825479862 >> v1:172.21.15.12:6789/0 conn(0x7f4ce40d00b0 legacy=0x7f4ce406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1967016666 0 0) 0x7f4cd0003520 con 0x7f4ce40d00b0 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce359e640 1 -- 172.21.15.12:0/825479862 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40d0cf0 con 0x7f4ce406ce00 2024-03-08T08:29:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.581+0000 7f4ce2d9d640 1 --2- 172.21.15.12:0/825479862 >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 0x7f4ce40cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce359e640 1 --2- 172.21.15.12:0/825479862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 0x7f4ce40cbf80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009d90 tx=0x7f4cd402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb23ce176aec4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4039070 con 0x7f4ce406ce00 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd402cc50 con 0x7f4ce406ce00 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce1d9b640 1 -- 172.21.15.12:0/825479862 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4034440 con 0x7f4ce406ce00 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 -- 172.21.15.12:0/825479862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 msgr2=0x7f4ce40cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 --2- 172.21.15.12:0/825479862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 0x7f4ce40cbf80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009d90 tx=0x7f4cd402bea0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 -- 172.21.15.12:0/825479862 shutdown_connections 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 --2- 172.21.15.12:0/825479862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce406ce00 0x7f4ce40cbf80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 --2- 172.21.15.12:0/825479862 >> v2:172.21.15.156:3300/0 conn(0x7f4ce409a580 0x7f4ce40cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 -- 172.21.15.12:0/825479862 >> 172.21.15.12:0/825479862 conn(0x7f4ce40b3420 msgr2=0x7f4ce40b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 -- 172.21.15.12:0/825479862 shutdown_connections 2024-03-08T08:29:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.582+0000 7f4ce918c640 1 -- 172.21.15.12:0/825479862 wait complete. 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.584+0000 7f4ce918c640 1 Processor -- start 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.584+0000 7f4ce918c640 1 -- start start 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce409a580 0x7f4ce40b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 0x7f4ce40ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce406c120 con 0x7f4ce406ce00 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40d12b0 con 0x7f4ce40d00b0 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce406c2a0 con 0x7f4ce409a580 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51884/0 (socket says 172.21.15.12:51884) 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 -- 172.21.15.12:0/2773633333 learned_addr learned my addr 172.21.15.12:0/2773633333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce359e640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 0x7f4ce40ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce259c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce409a580 0x7f4ce40b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 -- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce409a580 msgr2=0x7f4ce40b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce409a580 0x7f4ce40b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 -- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 msgr2=0x7f4ce40ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 0x7f4ce40ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 -- 172.21.15.12:0/2773633333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40b45f0 con 0x7f4ce406ce00 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce359e640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 0x7f4ce40ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:52.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce2d9d640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ebe0 tx=0x7f4cdc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc017070 con 0x7f4ce406ce00 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc00cc30 con 0x7f4ce406ce00 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce40b48d0 con 0x7f4ce406ce00 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc013470 con 0x7f4ce406ce00 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.585+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce40b4e90 con 0x7f4ce406ce00 2024-03-08T08:29:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.586+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce40749c0 con 0x7f4ce406ce00 2024-03-08T08:29:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.587+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cdc01a070 con 0x7f4ce406ce00 2024-03-08T08:29:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.587+0000 7f4cdb7fe640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 0x7f4cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.587+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cdc08a480 con 0x7f4ce406ce00 2024-03-08T08:29:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.587+0000 7f4ce259c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 0x7f4cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.588+0000 7f4ce259c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 0x7f4cc406d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4070b50 tx=0x7f4ccc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.592+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4cdc059390 con 0x7f4ce406ce00 2024-03-08T08:29:52.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.773+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ce40b42a0 con 0x7f4ce406ce00 2024-03-08T08:29:52.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.774+0000 7f4cdb7fe640 1 -- 172.21.15.12:0/2773633333 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f4cdc058c90 con 0x7f4ce406ce00 2024-03-08T08:29:52.776 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:52.776 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":21,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:07.231031+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.776+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 msgr2=0x7f4cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.776+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 0x7f4cc406d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4070b50 tx=0x7f4ccc009040 comp rx=0 tx=0).stop 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.776+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 msgr2=0x7f4ce4153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ebe0 tx=0x7f4cdc00c520 comp rx=0 tx=0).stop 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 shutdown_connections 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4cc406b4d0 0x7f4cc406d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ce40d00b0 0x7f4ce40ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ce409a580 0x7f4ce40b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 --2- 172.21.15.12:0/2773633333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce406ce00 0x7f4ce4153d80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 >> 172.21.15.12:0/2773633333 conn(0x7f4ce40b3420 msgr2=0x7f4ce40b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 shutdown_connections 2024-03-08T08:29:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.777+0000 7f4ce918c640 1 -- 172.21.15.12:0/2773633333 wait complete. 2024-03-08T08:29:52.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:29:03.767877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:29:04.567625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:29:52.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:29:52.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.892+0000 7fd89a86b640 1 Processor -- start 2024-03-08T08:29:52.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.892+0000 7fd89a86b640 1 -- start start 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd89a86b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 0x7fd8940b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd89a86b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd89406c160 0x7fd89406c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd89a86b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940d1d90 con 0x7fd894161260 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd89a86b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8940b9ac0 con 0x7fd89406c160 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd89a86b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89406caf0 con 0x7fd8940b6b30 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd89406c160 0x7fd89406c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd89406c160 0x7fd89406c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51900/0 (socket says 172.21.15.12:51900) 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 -- 172.21.15.12:0/1946471289 learned_addr learned my addr 172.21.15.12:0/1946471289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:52.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd8937fe640 1 --2- 172.21.15.12:0/1946471289 >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 0x7fd8940b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 -- 172.21.15.12:0/1946471289 >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 msgr2=0x7fd8940b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 --2- 172.21.15.12:0/1946471289 >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 0x7fd8940b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 -- 172.21.15.12:0/1946471289 >> v1:172.21.15.156:6789/0 conn(0x7fd894161260 legacy=0x7fd8940b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd898e08640 1 -- 172.21.15.12:0/1946471289 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940d2070 con 0x7fd89406c160 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.893+0000 7fd8937fe640 1 --2- 172.21.15.12:0/1946471289 >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 0x7fd8940b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:52.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.894+0000 7fd898e08640 1 --2- 172.21.15.12:0/1946471289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd89406c160 0x7fd89406c5b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd8940d3a30 tx=0x7fd88c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8293d7cb6663bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.894+0000 7fd892ffd640 1 -- 172.21.15.12:0/1946471289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c02ca80 con 0x7fd89406c160 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.894+0000 7fd892ffd640 1 -- 172.21.15.12:0/1946471289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c02cc20 con 0x7fd89406c160 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.894+0000 7fd892ffd640 1 -- 172.21.15.12:0/1946471289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0343f0 con 0x7fd89406c160 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 -- 172.21.15.12:0/1946471289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd89406c160 msgr2=0x7fd89406c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 --2- 172.21.15.12:0/1946471289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd89406c160 0x7fd89406c5b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd8940d3a30 tx=0x7fd88c02bee0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 -- 172.21.15.12:0/1946471289 shutdown_connections 2024-03-08T08:29:52.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 --2- 172.21.15.12:0/1946471289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd89406c160 0x7fd89406c5b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 --2- 172.21.15.12:0/1946471289 >> v2:172.21.15.156:3300/0 conn(0x7fd8940b6b30 0x7fd8940b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 -- 172.21.15.12:0/1946471289 >> 172.21.15.12:0/1946471289 conn(0x7fd894072930 msgr2=0x7fd8940d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 -- 172.21.15.12:0/1946471289 shutdown_connections 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.895+0000 7fd89a86b640 1 -- 172.21.15.12:0/1946471289 wait complete. 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.896+0000 7fd89a86b640 1 Processor -- start 2024-03-08T08:29:52.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.896+0000 7fd89a86b640 1 -- start start 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd89406c160 0x7fd8940c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8940b6b30 0x7fd89414b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940b2ec0 con 0x7fd894161260 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940d2630 con 0x7fd89406c160 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd89a86b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940d2e30 con 0x7fd8940b6b30 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd898e08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd898e08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51916/0 (socket says 172.21.15.12:51916) 2024-03-08T08:29:52.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd898e08640 1 -- 172.21.15.12:0/3629676928 learned_addr learned my addr 172.21.15.12:0/3629676928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd8937fe640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8940b6b30 0x7fd89414b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.897+0000 7fd893fff640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd89406c160 0x7fd8940c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 -- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8940b6b30 msgr2=0x7fd89414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8940b6b30 0x7fd89414b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 -- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd89406c160 msgr2=0x7fd8940c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd89406c160 0x7fd8940c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 -- 172.21.15.12:0/3629676928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940c7830 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd898e08640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd88c036040 tx=0x7fd88c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c043020 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c03e450 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c03e860 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8940cbe20 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.898+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940cc820 con 0x7fd894161260 2024-03-08T08:29:52.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.899+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8940cc340 con 0x7fd894161260 2024-03-08T08:29:52.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.899+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd88c046020 con 0x7fd894161260 2024-03-08T08:29:52.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.900+0000 7fd890ff9640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 0x7fd88006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:52.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.900+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd88c0ac650 con 0x7fd894161260 2024-03-08T08:29:52.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.900+0000 7fd893fff640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 0x7fd88006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:52.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.900+0000 7fd893fff640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 0x7fd88006d9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd888006fd0 tx=0x7fd888009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:52.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:52.904+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd8940cc340 con 0x7fd894161260 2024-03-08T08:29:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.083+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8940d3260 con 0x7fd894161260 2024-03-08T08:29:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.083+0000 7fd890ff9640 1 -- 172.21.15.12:0/3629676928 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd88c04d070 con 0x7fd894161260 2024-03-08T08:29:53.084 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:29:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 msgr2=0x7fd88006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 0x7fd88006d9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd888006fd0 tx=0x7fd888009060 comp rx=0 tx=0).stop 2024-03-08T08:29:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 msgr2=0x7fd8940c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd88c036040 tx=0x7fd88c004a40 comp rx=0 tx=0).stop 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 shutdown_connections 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd88006b500 0x7fd88006d9c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd894161260 0x7fd8940c72f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8940b6b30 0x7fd89414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 --2- 172.21.15.12:0/3629676928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd89406c160 0x7fd8940c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 >> 172.21.15.12:0/3629676928 conn(0x7fd894072930 msgr2=0x7fd8940b6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 shutdown_connections 2024-03-08T08:29:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.085+0000 7fd89a86b640 1 -- 172.21.15.12:0/3629676928 wait complete. 2024-03-08T08:29:53.097 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:29:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.200+0000 7f1f97a6f640 1 Processor -- start 2024-03-08T08:29:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.200+0000 7f1f97a6f640 1 -- start start 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f97a6f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1f9015dbc0 0x7f1f900b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f97a6f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900b7520 con 0x7f1f900b7080 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f97a6f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900b7800 con 0x7f1f900b9760 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f97a6f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f900b7ae0 con 0x7f1f9015dbc0 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f9500a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f900b7080 0x7f1f900d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56394/0 (socket says 172.21.15.12:56394) 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f9500a640 1 -- 172.21.15.12:0/1971819925 learned_addr learned my addr 172.21.15.12:0/1971819925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f9580b640 1 --2- 172.21.15.12:0/1971819925 >> v2:172.21.15.156:3300/0 conn(0x7f1f9015dbc0 0x7f1f900b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729129858 0 0) 0x7f1f900b7520 con 0x7f1f900b7080 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f7c003520 con 0x7f1f900b7080 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139823545 0 0) 0x7f1f7c003520 con 0x7f1f900b7080 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 >> v2:172.21.15.156:3300/0 conn(0x7f1f9015dbc0 msgr2=0x7f1f900b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 --2- 172.21.15.12:0/1971819925 >> v2:172.21.15.156:3300/0 conn(0x7f1f9015dbc0 0x7f1f900b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 >> v1:172.21.15.156:6789/0 conn(0x7f1f900b9760 legacy=0x7f1f900b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900b7dc0 con 0x7f1f900b7080 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.201+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f1f80001670 con 0x7f1f900b7080 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 >> v1:172.21.15.12:6789/0 conn(0x7f1f900b7080 legacy=0x7f1f900d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0039b0 0x7f1f7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f7c004800 0x7f1f7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900b7ae0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c00c060 con 0x7f1f7c0039b0 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c00c260 con 0x7f1f7c004800 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9580b640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0039b0 0x7f1f7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9500a640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f7c004800 0x7f1f7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 -- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f7c004800 msgr2=0x7f1f7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f7c004800 0x7f1f7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 -- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0039b0 msgr2=0x7f1f7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0039b0 0x7f1f7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9600c640 1 -- 172.21.15.12:0/1971819925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c003520 con 0x7f1f7c0051c0 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.202+0000 7f1f9500a640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f7c004800 0x7f1f7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f9600c640 1 -- 172.21.15.12:0/1971819925 reap_dead start 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f9600c640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f7c00b9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1f88001e50 tx=0x7f1f7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe49efa557368e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88001a70 con 0x7f1f7c0051c0 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f88003bb0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f94809640 1 -- 172.21.15.12:0/1971819925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f880050d0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f97a6f640 1 -- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 msgr2=0x7f1f7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f7c00b9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1f88001e50 tx=0x7f1f7c005830 comp rx=0 tx=0).stop 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f97a6f640 1 -- 172.21.15.12:0/1971819925 shutdown_connections 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/1971819925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f7c00b9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.203+0000 7f1f97a6f640 1 -- 172.21.15.12:0/1971819925 >> 172.21.15.12:0/1971819925 conn(0x7f1f90072720 msgr2=0x7f1f90072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.204+0000 7f1f97a6f640 1 -- 172.21.15.12:0/1971819925 shutdown_connections 2024-03-08T08:29:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.204+0000 7f1f97a6f640 1 -- 172.21.15.12:0/1971819925 wait complete. 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 Processor -- start 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 -- start start 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f900cbad0 0x7f1f90157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f901580c0 0x7f1f9015f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900b6b40 con 0x7f1f7c0051c0 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9015dfa0 con 0x7f1f900cbad0 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f97a6f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900b6cc0 con 0x7f1f901580c0 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.205+0000 7f1f9580b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51938/0 (socket says 172.21.15.12:51938) 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 -- 172.21.15.12:0/2018126436 learned_addr learned my addr 172.21.15.12:0/2018126436 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 -- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f901580c0 msgr2=0x7f1f9015f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f901580c0 0x7f1f9015f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 -- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f900cbad0 msgr2=0x7f1f90157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f900cbad0 0x7f1f90157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 -- 172.21.15.12:0/2018126436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9015e580 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f9580b640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0118e0 tx=0x7f1f8c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c00faf0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c00fc90 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9015e860 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c014440 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.206+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9015ee20 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.207+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f900cf400 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.208+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f8c012040 con 0x7f1f7c0051c0 2024-03-08T08:29:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.208+0000 7f1f867fc640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 0x7f1f7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.208+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f8c08a8a0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.208+0000 7f1f9500a640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 0x7f1f7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.208+0000 7f1f9500a640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 0x7f1f7406d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000f60 tx=0x7f1f80001df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.212+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f8c019030 con 0x7f1f7c0051c0 2024-03-08T08:29:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.398+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f900749c0 con 0x7f1f7c0051c0 2024-03-08T08:29:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.398+0000 7f1f867fc640 1 -- 172.21.15.12:0/2018126436 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f8c059160 con 0x7f1f7c0051c0 2024-03-08T08:29:53.399 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:29:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 msgr2=0x7f1f7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 0x7f1f7406d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000f60 tx=0x7f1f80001df0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 msgr2=0x7f1f900cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0118e0 tx=0x7f1f8c011da0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 shutdown_connections 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1f7406b500 0x7f1f7406d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f901580c0 0x7f1f9015f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f900cbad0 0x7f1f90157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 --2- 172.21.15.12:0/2018126436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c0051c0 0x7f1f900cd600 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 >> 172.21.15.12:0/2018126436 conn(0x7f1f90072720 msgr2=0x7f1f900d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 shutdown_connections 2024-03-08T08:29:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.400+0000 7f1f97a6f640 1 -- 172.21.15.12:0/2018126436 wait complete. 2024-03-08T08:29:53.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:29:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.515+0000 7fd36dbf4640 1 Processor -- start 2024-03-08T08:29:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.515+0000 7fd36dbf4640 1 -- start start 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd36dbf4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd36815dec0 0x7fd368161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd36dbf4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3680b6be0 con 0x7fd3680d0960 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd36dbf4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36815e2a0 con 0x7fd368161ed0 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd36dbf4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680d2140 con 0x7fd36815dec0 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd367fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd3680d0960 0x7fd3680d25d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56410/0 (socket says 172.21.15.12:56410) 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd367fff640 1 -- 172.21.15.12:0/4010291224 learned_addr learned my addr 172.21.15.12:0/4010291224 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3677fe640 1 --2- 172.21.15.12:0/4010291224 >> v2:172.21.15.156:3300/0 conn(0x7fd36815dec0 0x7fd368161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268054504 0 0) 0x7fd360001490 con 0x7fd3680d0960 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354003520 con 0x7fd3680d0960 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086420752 0 0) 0x7fd354003520 con 0x7fd3680d0960 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 >> v2:172.21.15.156:3300/0 conn(0x7fd36815dec0 msgr2=0x7fd368161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 --2- 172.21.15.12:0/4010291224 >> v2:172.21.15.156:3300/0 conn(0x7fd36815dec0 0x7fd368161990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 >> v1:172.21.15.156:6789/0 conn(0x7fd368161ed0 legacy=0x7fd3680d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368162350 con 0x7fd3680d0960 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fd360001bc0 con 0x7fd3680d0960 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 >> v1:172.21.15.12:6789/0 conn(0x7fd3680d0960 legacy=0x7fd3680d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3540039b0 0x7fd354003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd354004800 0x7fd354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 0x7fd35400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680d2140 con 0x7fd3540051c0 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35400c060 con 0x7fd3540039b0 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.516+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35400c260 con 0x7fd354004800 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 0x7fd35400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd3677fe640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3540039b0 0x7fd354003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd366ffd640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd354004800 0x7fd354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 -- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd354004800 msgr2=0x7fd354004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd354004800 0x7fd354004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 -- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3540039b0 msgr2=0x7fd354003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3540039b0 0x7fd354003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 -- 172.21.15.12:0/4010291224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354003520 con 0x7fd3540051c0 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.517+0000 7fd367fff640 1 -- 172.21.15.12:0/4010291224 reap_dead start 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd367fff640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 0x7fd35400b9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd36000aa50 tx=0x7fd354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b30aab70373e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360038070 con 0x7fd3540051c0 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360033650 con 0x7fd3540051c0 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd3667fc640 1 -- 172.21.15.12:0/4010291224 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000a670 con 0x7fd3540051c0 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 -- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 msgr2=0x7fd35400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 0x7fd35400b9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd36000aa50 tx=0x7fd354005830 comp rx=0 tx=0).stop 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 -- 172.21.15.12:0/4010291224 shutdown_connections 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/4010291224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3540051c0 0x7fd35400b9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 -- 172.21.15.12:0/4010291224 >> 172.21.15.12:0/4010291224 conn(0x7fd36806c240 msgr2=0x7fd36806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 -- 172.21.15.12:0/4010291224 shutdown_connections 2024-03-08T08:29:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.518+0000 7fd36dbf4640 1 -- 172.21.15.12:0/4010291224 wait complete. 2024-03-08T08:29:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.519+0000 7fd36dbf4640 1 Processor -- start 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 -- start start 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3540051c0 0x7fd3680cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 0x7fd3680cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680b94f0 con 0x7fd3680ccd40 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680d1dd0 con 0x7fd3680cb0c0 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd36dbf4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680b9670 con 0x7fd3540051c0 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd367fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd367fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51964/0 (socket says 172.21.15.12:51964) 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.520+0000 7fd367fff640 1 -- 172.21.15.12:0/251678466 learned_addr learned my addr 172.21.15.12:0/251678466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd366ffd640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 0x7fd3680cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd3677fe640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3540051c0 0x7fd3680cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 -- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3540051c0 msgr2=0x7fd3680cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3540051c0 0x7fd3680cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 -- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 msgr2=0x7fd3680cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 0x7fd3680cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 -- 172.21.15.12:0/251678466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680cb870 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd366ffd640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 0x7fd3680cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd367fff640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd36000a0b0 tx=0x7fd360001f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360038040 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360042070 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3680cbb50 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360001490 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.521+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36815f580 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.522+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680749c0 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.522+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd360033650 con 0x7fd3680ccd40 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.523+0000 7fd35ffff640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 0x7fd34806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.523+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd3600abd30 con 0x7fd3680ccd40 2024-03-08T08:29:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.523+0000 7fd3677fe640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 0x7fd34806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.523+0000 7fd3677fe640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 0x7fd34806d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3680d16a0 tx=0x7fd358008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.527+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd3680b6be0 con 0x7fd3680ccd40 2024-03-08T08:29:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.709+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3680ceeb0 con 0x7fd3680ccd40 2024-03-08T08:29:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.709+0000 7fd35ffff640 1 -- 172.21.15.12:0/251678466 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd36007a540 con 0x7fd3680ccd40 2024-03-08T08:29:53.710 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:29:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 msgr2=0x7fd34806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 0x7fd34806d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3680d16a0 tx=0x7fd358008040 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 msgr2=0x7fd3680cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd36000a0b0 tx=0x7fd360001f90 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 shutdown_connections 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd34806b500 0x7fd34806d9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3680ccd40 0x7fd3680cd1f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3680cb0c0 0x7fd3680cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 --2- 172.21.15.12:0/251678466 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3540051c0 0x7fd3680cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 >> 172.21.15.12:0/251678466 conn(0x7fd36806c240 msgr2=0x7fd36815c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 shutdown_connections 2024-03-08T08:29:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.711+0000 7fd36dbf4640 1 -- 172.21.15.12:0/251678466 wait complete. 2024-03-08T08:29:53.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:53.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 Processor -- start 2024-03-08T08:29:53.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 -- start start 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe1c409b150 0x7fe1c40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1c4099ca0 0x7fe1c409a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c409a5f0 con 0x7fe1c415dff0 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c40d0910 con 0x7fe1c4099ca0 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c91a1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c40cca40 con 0x7fe1c409b150 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.824+0000 7fe1c359e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1c4099ca0 0x7fe1c409a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1c4099ca0 0x7fe1c409a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51974/0 (socket says 172.21.15.12:51974) 2024-03-08T08:29:53.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 -- 172.21.15.12:0/4262436411 learned_addr learned my addr 172.21.15.12:0/4262436411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1bbfff640 1 --2- 172.21.15.12:0/4262436411 >> v2:172.21.15.156:3300/0 conn(0x7fe1c409b150 0x7fe1c40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 -- 172.21.15.12:0/4262436411 >> v2:172.21.15.156:3300/0 conn(0x7fe1c409b150 msgr2=0x7fe1c40cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 --2- 172.21.15.12:0/4262436411 >> v2:172.21.15.156:3300/0 conn(0x7fe1c409b150 0x7fe1c40cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 -- 172.21.15.12:0/4262436411 >> v1:172.21.15.156:6789/0 conn(0x7fe1c415dff0 legacy=0x7fe1c40cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.825+0000 7fe1c359e640 1 -- 172.21.15.12:0/4262436411 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40ccd20 con 0x7fe1c4099ca0 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c359e640 1 --2- 172.21.15.12:0/4262436411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c409a0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4072da0 tx=0x7fe1bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9836146dac54599 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c259c640 1 -- 172.21.15.12:0/4262436411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc02c9f0 con 0x7fe1c4099ca0 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c259c640 1 -- 172.21.15.12:0/4262436411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc02cb90 con 0x7fe1c4099ca0 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c259c640 1 -- 172.21.15.12:0/4262436411 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc0343f0 con 0x7fe1c4099ca0 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 -- 172.21.15.12:0/4262436411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 msgr2=0x7fe1c409a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/4262436411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c409a0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4072da0 tx=0x7fe1bc02be50 comp rx=0 tx=0).stop 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 -- 172.21.15.12:0/4262436411 shutdown_connections 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/4262436411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c409a0b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/4262436411 >> v2:172.21.15.156:3300/0 conn(0x7fe1c409b150 0x7fe1c40cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 -- 172.21.15.12:0/4262436411 >> 172.21.15.12:0/4262436411 conn(0x7fe1c40d1900 msgr2=0x7fe1c40d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:53.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 -- 172.21.15.12:0/4262436411 shutdown_connections 2024-03-08T08:29:53.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.826+0000 7fe1c91a1640 1 -- 172.21.15.12:0/4262436411 wait complete. 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 Processor -- start 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 -- start start 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1c40c63b0 0x7fe1c415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1c40b4520 0x7fe1c40b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c406f7e0 con 0x7fe1c4099ca0 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40d05a0 con 0x7fe1c40b4520 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.828+0000 7fe1c91a1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c406f960 con 0x7fe1c40c63b0 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51984/0 (socket says 172.21.15.12:51984) 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 -- 172.21.15.12:0/2881929369 learned_addr learned my addr 172.21.15.12:0/2881929369 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c359e640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1c40b4520 0x7fe1c40b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1bbfff640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1c40c63b0 0x7fe1c415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 -- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1c40c63b0 msgr2=0x7fe1c415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1c40c63b0 0x7fe1c415af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 -- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1c40b4520 msgr2=0x7fe1c40b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1c40b4520 0x7fe1c40b6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 -- 172.21.15.12:0/2881929369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40c7990 con 0x7fe1c4099ca0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c2d9d640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400b3f0 tx=0x7fe1b400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4015020 con 0x7fe1c4099ca0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b4011830 con 0x7fe1c4099ca0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c40c8a70 con 0x7fe1c4099ca0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011c20 con 0x7fe1c4099ca0 2024-03-08T08:29:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.829+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c40c9030 con 0x7fe1c4099ca0 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.830+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c40c6790 con 0x7fe1c4099ca0 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.831+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1b4018020 con 0x7fe1c4099ca0 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.831+0000 7fe1bb7fe640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 0x7fe1a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.831+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe1b4089db0 con 0x7fe1c4099ca0 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.831+0000 7fe1bbfff640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 0x7fe1a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:53.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.831+0000 7fe1bbfff640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 0x7fe1a406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40ca530 tx=0x7fe1ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:53.835+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe1c40c6790 con 0x7fe1c4099ca0 2024-03-08T08:29:54.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.051+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c4071530 con 0x7fe1c4099ca0 2024-03-08T08:29:54.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.051+0000 7fe1bb7fe640 1 -- 172.21.15.12:0/2881929369 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7fe1b401f030 con 0x7fe1c4099ca0 2024-03-08T08:29:54.052 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:54.052 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 msgr2=0x7fe1a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 0x7fe1a406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40ca530 tx=0x7fe1ac008040 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 msgr2=0x7fe1c40c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400b3f0 tx=0x7fe1b400b8b0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 shutdown_connections 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fe1a406b500 0x7fe1a406d9c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1c40b4520 0x7fe1c40b6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1c40c63b0 0x7fe1c415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 --2- 172.21.15.12:0/2881929369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c4099ca0 0x7fe1c40c7380 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 >> 172.21.15.12:0/2881929369 conn(0x7fe1c40d1900 msgr2=0x7fe1c40d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 shutdown_connections 2024-03-08T08:29:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.053+0000 7fe1c91a1640 1 -- 172.21.15.12:0/2881929369 wait complete. 2024-03-08T08:29:54.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:54.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.165+0000 7f0272dfd640 1 Processor -- start 2024-03-08T08:29:54.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.165+0000 7f0272dfd640 1 -- start start 2024-03-08T08:29:54.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f0272dfd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f026c09a580 0x7f026c0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f0272dfd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c09a960 con 0x7f026c0d00b0 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f0272dfd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0d0730 con 0x7f026c06ce00 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f0272dfd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0d0a10 con 0x7f026c09a580 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f026bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f026c0d00b0 0x7f026c06c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56424/0 (socket says 172.21.15.12:56424) 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f026bfff640 1 -- 172.21.15.12:0/3597572607 learned_addr learned my addr 172.21.15.12:0/3597572607 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f0270b99640 1 --2- 172.21.15.12:0/3597572607 >> v2:172.21.15.156:3300/0 conn(0x7f026c09a580 0x7f026c0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001618502 0 0) 0x7f026c09a960 con 0x7f026c0d00b0 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.166+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0258003520 con 0x7f026c0d00b0 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 6779173 0 0) 0x7f0258003520 con 0x7f026c0d00b0 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 >> v2:172.21.15.156:3300/0 conn(0x7f026c09a580 msgr2=0x7f026c0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 --2- 172.21.15.12:0/3597572607 >> v2:172.21.15.156:3300/0 conn(0x7f026c09a580 0x7f026c0cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 >> v1:172.21.15.156:6789/0 conn(0x7f026c06ce00 legacy=0x7f026c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0d0cf0 con 0x7f026c0d00b0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f025c001670 con 0x7f026c0d00b0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 >> v1:172.21.15.12:6789/0 conn(0x7f026c0d00b0 legacy=0x7f026c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580039b0 0x7f0258003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 0x7f0258004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0258005080 0x7f025800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0d0a10 con 0x7f02580043b0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025800bfe0 con 0x7f02580039b0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025800c1e0 con 0x7f0258005080 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 0x7f0258004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f0270b99640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580039b0 0x7f0258003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f027139a640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0258005080 0x7f025800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 -- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0258005080 msgr2=0x7f025800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0258005080 0x7f025800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 -- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580039b0 msgr2=0x7f0258003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580039b0 0x7f0258003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f026bfff640 1 -- 172.21.15.12:0/3597572607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258003520 con 0x7f02580043b0 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.167+0000 7f027139a640 1 -- 172.21.15.12:0/3597572607 reap_dead start 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f026bfff640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 0x7f0258004880 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c290 tx=0x7f0258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad984dee1dea100d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c007d50 con 0x7f02580043b0 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c030070 con 0x7f02580043b0 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f026b7fe640 1 -- 172.21.15.12:0/3597572607 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c02cda0 con 0x7f02580043b0 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 -- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 msgr2=0x7f0258004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 0x7f0258004880 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c290 tx=0x7f0258005830 comp rx=0 tx=0).stop 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 -- 172.21.15.12:0/3597572607 shutdown_connections 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3597572607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02580043b0 0x7f0258004880 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 -- 172.21.15.12:0/3597572607 >> 172.21.15.12:0/3597572607 conn(0x7f026c0b3420 msgr2=0x7f026c0b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 -- 172.21.15.12:0/3597572607 shutdown_connections 2024-03-08T08:29:54.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.168+0000 7f0272dfd640 1 -- 172.21.15.12:0/3597572607 wait complete. 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.170+0000 7f0272dfd640 1 Processor -- start 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.170+0000 7f0272dfd640 1 -- start start 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.170+0000 7f0272dfd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580043b0 0x7f026c09c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.170+0000 7f0272dfd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f026c15dad0 0x7f026c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c06c120 con 0x7f026c09e880 2024-03-08T08:29:54.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0d12b0 con 0x7f02580043b0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c06c2a0 con 0x7f026c15dad0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51996/0 (socket says 172.21.15.12:51996) 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 -- 172.21.15.12:0/3217318807 learned_addr learned my addr 172.21.15.12:0/3217318807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0270b99640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580043b0 0x7f026c09c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f027139a640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f026c15dad0 0x7f026c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 -- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f026c15dad0 msgr2=0x7f026c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f026c15dad0 0x7f026c15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 -- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580043b0 msgr2=0x7f026c09c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580043b0 0x7f026c09c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 -- 172.21.15.12:0/3217318807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c09eff0 con 0x7f026c09e880 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f026bfff640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c3b0 tx=0x7f025c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c046070 con 0x7f026c09e880 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c004bf0 con 0x7f026c09e880 2024-03-08T08:29:54.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c09f2d0 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c007d50 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.171+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c1f3790 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.172+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c0749c0 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.173+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f025c007070 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.173+0000 7f02697fa640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 0x7f024c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.173+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f025c0ab3c0 con 0x7f026c09e880 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.173+0000 7f0270b99640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 0x7f024c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.174+0000 7f0270b99640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 0x7f024c06d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0264003c00 tx=0x7f026400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.178+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f025c079c60 con 0x7f026c09e880 2024-03-08T08:29:54.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.361+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f026c0b42a0 con 0x7f026c09e880 2024-03-08T08:29:54.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.362+0000 7f02697fa640 1 -- 172.21.15.12:0/3217318807 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16658 (secure 0 0 0) 0x7f025c07dd00 con 0x7f026c09e880 2024-03-08T08:29:54.364 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:53.376974+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 msgr2=0x7f024c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 0x7f024c06d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0264003c00 tx=0x7f026400b040 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 msgr2=0x7f026c0c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c3b0 tx=0x7f025c02e070 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 shutdown_connections 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f024c06b500 0x7f024c06d9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f026c15dad0 0x7f026c15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f026c09e880 0x7f026c0c7db0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 --2- 172.21.15.12:0/3217318807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02580043b0 0x7f026c09c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 >> 172.21.15.12:0/3217318807 conn(0x7f026c0b3420 msgr2=0x7f026c0b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 shutdown_connections 2024-03-08T08:29:54.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.364+0000 7f0272dfd640 1 -- 172.21.15.12:0/3217318807 wait complete. 2024-03-08T08:29:54.377 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-08T08:29:54.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.478+0000 7ffaab00e640 1 Processor -- start 2024-03-08T08:29:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.478+0000 7ffaab00e640 1 -- start start 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaab00e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffaa40d0a30 0x7ffaa40b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaab00e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffaa4161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaab00e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa4161fb0 con 0x7ffaa40d0e10 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaab00e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa4161d80 con 0x7ffaa40d0a30 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaab00e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40d1d60 con 0x7ffaa415df10 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa95ab640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffaa40d0e10 0x7ffaa40b54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56432/0 (socket says 172.21.15.12:56432) 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa95ab640 1 -- 172.21.15.12:0/654233599 learned_addr learned my addr 172.21.15.12:0/654233599 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/654233599 >> v2:172.21.15.156:3300/0 conn(0x7ffaa40d0a30 0x7ffaa40b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794502866 0 0) 0x7ffaa4161fb0 con 0x7ffaa40d0e10 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffaa4161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa94003520 con 0x7ffaa40d0e10 2024-03-08T08:29:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088280093 0 0) 0x7ffa94003520 con 0x7ffaa40d0e10 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 msgr2=0x7ffaa4161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffaa4161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 >> v2:172.21.15.156:3300/0 conn(0x7ffaa40d0a30 msgr2=0x7ffaa40b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 --2- 172.21.15.12:0/654233599 >> v2:172.21.15.156:3300/0 conn(0x7ffaa40d0a30 0x7ffaa40b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4162290 con 0x7ffaa40d0e10 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7ffa9c0019f0 con 0x7ffaa40d0e10 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.479+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 >> v1:172.21.15.12:6789/0 conn(0x7ffaa40d0e10 legacy=0x7ffaa40b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffa940039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa94003f30 0x7ffa94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94004d40 0x7ffa940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40d1d60 con 0x7ffa94003f30 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9400b9c0 con 0x7ffa94004d40 2024-03-08T08:29:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9400bb40 con 0x7ffaa415df10 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa94003f30 0x7ffa94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffa940039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa95ab640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94004d40 0x7ffa940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 -- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 msgr2=0x7ffa940039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa415df10 0x7ffa940039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 -- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94004d40 msgr2=0x7ffa940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94004d40 0x7ffa940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa3fff640 1 -- 172.21.15.12:0/654233599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94003520 con 0x7ffa94003f30 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.480+0000 7ffaa95ab640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94004d40 0x7ffa940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa94003f30 0x7ffa94004800 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009a90 tx=0x7ffa94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90925921adec4805 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90039070 con 0x7ffa94003f30 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9002c6b0 con 0x7ffa94003f30 2024-03-08T08:29:54.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaa37fe640 1 -- 172.21.15.12:0/654233599 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90009690 con 0x7ffa94003f30 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 -- 172.21.15.12:0/654233599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa94003f30 msgr2=0x7ffa94004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 --2- 172.21.15.12:0/654233599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa94003f30 0x7ffa94004800 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009a90 tx=0x7ffa94005830 comp rx=0 tx=0).stop 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaa95ab640 1 -- 172.21.15.12:0/654233599 reap_dead start 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 -- 172.21.15.12:0/654233599 shutdown_connections 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 -- 172.21.15.12:0/654233599 >> 172.21.15.12:0/654233599 conn(0x7ffaa406c710 msgr2=0x7ffaa406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 -- 172.21.15.12:0/654233599 shutdown_connections 2024-03-08T08:29:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.481+0000 7ffaab00e640 1 -- 172.21.15.12:0/654233599 wait complete. 2024-03-08T08:29:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.482+0000 7ffaab00e640 1 Processor -- start 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 -- start start 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 0x7ffaa41e85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa4159980 0x7ffaa4159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40d12a0 con 0x7ffaa4161300 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40d1fb0 con 0x7ffa94003f30 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaab00e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40d34e0 con 0x7ffaa4159980 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.483+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52020/0 (socket says 172.21.15.12:52020) 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 -- 172.21.15.12:0/3404157916 learned_addr learned my addr 172.21.15.12:0/3404157916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa95ab640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa4159980 0x7ffaa4159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 0x7ffaa41e85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 -- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa4159980 msgr2=0x7ffaa4159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa4159980 0x7ffaa4159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 -- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 msgr2=0x7ffaa41e85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 0x7ffaa41e85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 -- 172.21.15.12:0/3404157916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa41e9f50 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 0x7ffaa41e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa3fff640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009a90 tx=0x7ffa90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90039050 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa90045070 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa41e8eb0 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900044e0 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.484+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa41e9470 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.485+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa40749c0 con 0x7ffaa4161300 2024-03-08T08:29:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.486+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa90003850 con 0x7ffaa4161300 2024-03-08T08:29:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.486+0000 7ffaa17fa640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 0x7ffa8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.486+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffa900abb50 con 0x7ffaa4161300 2024-03-08T08:29:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.486+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 0x7ffa8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.486+0000 7ffaa8daa640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 0x7ffa8806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009c40 tx=0x7ffa98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.490+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffa9004e020 con 0x7ffaa4161300 2024-03-08T08:29:54.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.709+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaa40cd810 con 0x7ffaa4161300 2024-03-08T08:29:54.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.709+0000 7ffaa17fa640 1 -- 172.21.15.12:0/3404157916 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7ffa9007aa40 con 0x7ffaa4161300 2024-03-08T08:29:54.711 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:54.711 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:54.711 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 msgr2=0x7ffa8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 0x7ffa8806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009c40 tx=0x7ffa98008040 comp rx=0 tx=0).stop 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 msgr2=0x7ffaa415e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009a90 tx=0x7ffa90004730 comp rx=0 tx=0).stop 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 shutdown_connections 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7ffa8806b4d0 0x7ffa8806d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffaa4159980 0x7ffaa4159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffaa4161300 0x7ffaa415e5b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.711+0000 7ffaab00e640 1 --2- 172.21.15.12:0/3404157916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa94003f30 0x7ffaa41e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.712+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 >> 172.21.15.12:0/3404157916 conn(0x7ffaa406c710 msgr2=0x7ffaa40b6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.712+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 shutdown_connections 2024-03-08T08:29:54.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.712+0000 7ffaab00e640 1 -- 172.21.15.12:0/3404157916 wait complete. 2024-03-08T08:29:54.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:29:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.824+0000 7f8cf7772640 1 Processor -- start 2024-03-08T08:29:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.824+0000 7f8cf7772640 1 -- start start 2024-03-08T08:29:54.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf7772640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8cf00b5060 0x7f8cf00b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf7772640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf015c980 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf7772640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf015cc60 con 0x7f8cf01597c0 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf7772640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf015c760 con 0x7f8cf00b5060 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf4d0d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8cf006b810 0x7f8cf015fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56438/0 (socket says 172.21.15.12:56438) 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf4d0d640 1 -- 172.21.15.12:0/3411432279 learned_addr learned my addr 172.21.15.12:0/3411432279 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945707229 0 0) 0x7f8cf015c980 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8cf5d0f640 1 --2- 172.21.15.12:0/3411432279 >> v2:172.21.15.156:3300/0 conn(0x7f8cf00b5060 0x7f8cf00b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cdc003520 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938123920 0 0) 0x7f8cdc003520 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 >> v2:172.21.15.156:3300/0 conn(0x7f8cf00b5060 msgr2=0x7f8cf00b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 --2- 172.21.15.12:0/3411432279 >> v2:172.21.15.156:3300/0 conn(0x7f8cf00b5060 0x7f8cf00b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 >> v1:172.21.15.156:6789/0 conn(0x7f8cf01597c0 legacy=0x7f8cf015c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf015cf40 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f8ce0001670 con 0x7f8cf006b810 2024-03-08T08:29:54.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 >> v1:172.21.15.12:6789/0 conn(0x7f8cf006b810 legacy=0x7f8cf015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cdc0039b0 0x7f8cdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.825+0000 7f8ce7fff640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cdc0052e0 0x7f8cdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf015c760 con 0x7f8cdc004800 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc00c1b0 con 0x7f8cdc0052e0 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc00c3b0 con 0x7f8cdc0039b0 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf550e640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cdc0039b0 0x7f8cdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf5d0f640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cdc0052e0 0x7f8cdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 -- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cdc0039b0 msgr2=0x7f8cdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cdc0039b0 0x7f8cdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 -- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cdc0052e0 msgr2=0x7f8cdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cdc0052e0 0x7f8cdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf4d0d640 1 -- 172.21.15.12:0/3411432279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc003520 con 0x7f8cdc004800 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.826+0000 7f8cf5d0f640 1 -- 172.21.15.12:0/3411432279 reap_dead start 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cdc004c80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0001eb0 tx=0x7f8cdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c08460a564ffd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0007d30 con 0x7f8cdc004800 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce0030070 con 0x7f8cdc004800 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8ce7fff640 1 -- 172.21.15.12:0/3411432279 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0005d70 con 0x7f8cdc004800 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 -- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 msgr2=0x7f8cdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cdc004c80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0001eb0 tx=0x7f8cdc005830 comp rx=0 tx=0).stop 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 -- 172.21.15.12:0/3411432279 shutdown_connections 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 --2- 172.21.15.12:0/3411432279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cdc004c80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 -- 172.21.15.12:0/3411432279 >> 172.21.15.12:0/3411432279 conn(0x7f8cf00b5ca0 msgr2=0x7f8cf00b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:54.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 -- 172.21.15.12:0/3411432279 shutdown_connections 2024-03-08T08:29:54.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.827+0000 7f8cf7772640 1 -- 172.21.15.12:0/3411432279 wait complete. 2024-03-08T08:29:54.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.828+0000 7f8cf7772640 1 Processor -- start 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 -- start start 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cf013dc90 0x7f8cf013e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 0x7f8cf013ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00b54b0 con 0x7f8cdc004800 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf015f980 con 0x7f8cf013e900 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf7772640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00b5630 con 0x7f8cf013dc90 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf550e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf550e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52044/0 (socket says 172.21.15.12:52044) 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.829+0000 7f8cf550e640 1 -- 172.21.15.12:0/2262407349 learned_addr learned my addr 172.21.15.12:0/2262407349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf5d0f640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 0x7f8cf013ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cf013dc90 0x7f8cf013e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 -- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cf013dc90 msgr2=0x7f8cf013e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cf013dc90 0x7f8cf013e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 -- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 msgr2=0x7f8cf013ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 0x7f8cf013ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 -- 172.21.15.12:0/2262407349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf013e630 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf5d0f640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 0x7f8cf013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf550e640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00f620 tx=0x7f8cec00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0052e0 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec018070 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec003810 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf013f2c0 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.830+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf013f830 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.831+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf00749c0 con 0x7f8cdc004800 2024-03-08T08:29:54.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.831+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cec0039b0 con 0x7f8cdc004800 2024-03-08T08:29:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.832+0000 7f8ce5ffb640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.832+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8cec08a080 con 0x7f8cdc004800 2024-03-08T08:29:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.832+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.832+0000 7f8cf4d0d640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 0x7f8cd406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0034000 tx=0x7f8ce0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:54.836+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cec058e50 con 0x7f8cdc004800 2024-03-08T08:29:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.053+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cf01e4ea0 con 0x7f8cdc004800 2024-03-08T08:29:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.054+0000 7f8ce5ffb640 1 -- 172.21.15.12:0/2262407349 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f8cec058750 con 0x7f8cdc004800 2024-03-08T08:29:55.055 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:55.055 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3274091859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3274091859},{"type":"v1","addr":"172.21.15.156:6834","nonce":3274091859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/735315116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":735315116},{"type":"v1","addr":"172.21.15.156:6835","nonce":735315116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6836/3452281311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3452281311},{"type":"v1","addr":"172.21.15.12:6836","nonce":3452281311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:29:05.596568+0000","modified":"2024-03-08T08:29:07.598513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/3547610523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6835","nonce":3547610523},{"type":"v1","addr":"172.21.15.12:6837","nonce":3547610523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T08:29:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 msgr2=0x7f8cd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 0x7f8cd406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0034000 tx=0x7f8ce0009030 comp rx=0 tx=0).stop 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 msgr2=0x7f8cf015b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00f620 tx=0x7f8cec00fae0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 shutdown_connections 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.055+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cf013e900 0x7f8cf013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.056+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8cf013dc90 0x7f8cf013e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.056+0000 7f8cf7772640 1 --2- 172.21.15.12:0/2262407349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc004800 0x7f8cf015b100 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.056+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 >> 172.21.15.12:0/2262407349 conn(0x7f8cf00b5ca0 msgr2=0x7f8cf015dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.056+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 shutdown_connections 2024-03-08T08:29:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.056+0000 7f8cf7772640 1 -- 172.21.15.12:0/2262407349 wait complete. 2024-03-08T08:29:55.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:29:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.168+0000 7fec2045c640 1 Processor -- start 2024-03-08T08:29:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.168+0000 7fec2045c640 1 -- start start 2024-03-08T08:29:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec1815dbd0 0x7fec181617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 0x7fec180ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 0x7fec180b5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec181626f0 con 0x7fec1815dbd0 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec18162130 con 0x7fec180cec60 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec2045c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18162410 con 0x7fec180b94e0 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec1e1f8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec1815dbd0 0x7fec181617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec1e1f8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec1815dbd0 0x7fec181617c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52052/0 (socket says 172.21.15.12:52052) 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.169+0000 7fec1e1f8640 1 -- 172.21.15.12:0/153931333 learned_addr learned my addr 172.21.15.12:0/153931333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e9f9640 1 --2- 172.21.15.12:0/153931333 >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 0x7fec180b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 -- 172.21.15.12:0/153931333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 msgr2=0x7fec180ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1d9f7640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 0x7fec180ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 0x7fec180ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 -- 172.21.15.12:0/153931333 >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 msgr2=0x7fec180b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/153931333 >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 0x7fec180b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 -- 172.21.15.12:0/153931333 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180d1c80 con 0x7fec1815dbd0 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e9f9640 1 --2- 172.21.15.12:0/153931333 >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 0x7fec180b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1d9f7640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 0x7fec180ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.170+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec1815dbd0 0x7fec181617c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fec180cf830 tx=0x7fec1002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6f34cefe2bd47 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec1d1f6640 1 -- 172.21.15.12:0/153931333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10035020 con 0x7fec1815dbd0 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec1d1f6640 1 -- 172.21.15.12:0/153931333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1002d040 con 0x7fec1815dbd0 2024-03-08T08:29:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec1d1f6640 1 -- 172.21.15.12:0/153931333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10031a80 con 0x7fec1815dbd0 2024-03-08T08:29:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 -- 172.21.15.12:0/153931333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec1815dbd0 msgr2=0x7fec181617c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec1815dbd0 0x7fec181617c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fec180cf830 tx=0x7fec1002b900 comp rx=0 tx=0).stop 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 -- 172.21.15.12:0/153931333 shutdown_connections 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 --2- 172.21.15.12:0/153931333 >> v2:172.21.15.156:3300/0 conn(0x7fec180cec60 0x7fec180b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec180b94e0 0x7fec180ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 --2- 172.21.15.12:0/153931333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec1815dbd0 0x7fec181617c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 -- 172.21.15.12:0/153931333 >> 172.21.15.12:0/153931333 conn(0x7fec180d3cc0 msgr2=0x7fec180d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 -- 172.21.15.12:0/153931333 shutdown_connections 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.171+0000 7fec2045c640 1 -- 172.21.15.12:0/153931333 wait complete. 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.172+0000 7fec2045c640 1 Processor -- start 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.172+0000 7fec2045c640 1 -- start start 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec180b94e0 0x7fec18157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec1815dbd0 0x7fec180b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18161d00 con 0x7fec180cec60 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1815dfb0 con 0x7fec180b94e0 2024-03-08T08:29:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec2045c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18161e80 con 0x7fec1815dbd0 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52066/0 (socket says 172.21.15.12:52066) 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 -- 172.21.15.12:0/3944882924 learned_addr learned my addr 172.21.15.12:0/3944882924 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec180b94e0 0x7fec18157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1e9f9640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec1815dbd0 0x7fec180b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 -- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec1815dbd0 msgr2=0x7fec180b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec1815dbd0 0x7fec180b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 -- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec180b94e0 msgr2=0x7fec18157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec180b94e0 0x7fec18157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.173+0000 7fec1d9f7640 1 -- 172.21.15.12:0/3944882924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180b5a20 con 0x7fec180cec60 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec1d9f7640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c7c0 tx=0x7fec0c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c015070 con 0x7fec180cec60 2024-03-08T08:29:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c007a30 con 0x7fec180cec60 2024-03-08T08:29:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec180b5d00 con 0x7fec180cec60 2024-03-08T08:29:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c007e00 con 0x7fec180cec60 2024-03-08T08:29:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.174+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1815eff0 con 0x7fec180cec60 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.175+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1815f2b0 con 0x7fec180cec60 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.175+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec0c018070 con 0x7fec180cec60 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.176+0000 7fec16ffd640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 0x7fec0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.176+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fec0c08a100 con 0x7fec180cec60 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.176+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 0x7fec0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.176+0000 7fec1e1f8640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 0x7fec0406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec10036000 tx=0x7fec1003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.180+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec0c058ff0 con 0x7fec180cec60 2024-03-08T08:29:55.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.363+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec180749c0 con 0x7fec180cec60 2024-03-08T08:29:55.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.364+0000 7fec16ffd640 1 -- 172.21.15.12:0/3944882924 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16658 (secure 0 0 0) 0x7fec0c0588f0 con 0x7fec180cec60 2024-03-08T08:29:55.365 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:29:55.365 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:29:54.380995+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:29:03.767877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T08:29:04.567625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:29:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 msgr2=0x7fec0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 0x7fec0406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec10036000 tx=0x7fec1003c040 comp rx=0 tx=0).stop 2024-03-08T08:29:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 msgr2=0x7fec1815eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c7c0 tx=0x7fec0c00cc80 comp rx=0 tx=0).stop 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 shutdown_connections 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fec0406b4d0 0x7fec0406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fec1815dbd0 0x7fec180b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec180cec60 0x7fec1815eab0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 --2- 172.21.15.12:0/3944882924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec180b94e0 0x7fec18157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 >> 172.21.15.12:0/3944882924 conn(0x7fec180d3cc0 msgr2=0x7fec180c8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 shutdown_connections 2024-03-08T08:29:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.366+0000 7fec2045c640 1 -- 172.21.15.12:0/3944882924 wait complete. 2024-03-08T08:29:55.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-08T08:29:55.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.479+0000 7fbab0d9a640 1 Processor -- start 2024-03-08T08:29:55.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.479+0000 7fbab0d9a640 1 -- start start 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbab0d9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbaac0d1110 0x7fbaac0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbab0d9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d31c0 0x7fbaac0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbab0d9a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaac15e1a0 con 0x7fbaac15ddc0 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbab0d9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaac0b9030 con 0x7fbaac0d1110 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbab0d9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac0d1d20 con 0x7fbaac0d31c0 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbaa9d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbaac0d1110 0x7fbaac0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbaaa59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbaac15ddc0 0x7fbaac0d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56450/0 (socket says 172.21.15.12:56450) 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.480+0000 7fbaaa59c640 1 -- 172.21.15.12:0/1408471552 learned_addr learned my addr 172.21.15.12:0/1408471552 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/1408471552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d31c0 msgr2=0x7fbaac0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa9d9b640 1 --2- 172.21.15.12:0/1408471552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d31c0 0x7fbaac0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/1408471552 >> v1:172.21.15.12:6789/0 conn(0x7fbaac15ddc0 legacy=0x7fbaac0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/1408471552 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac0b8790 con 0x7fbaac0d1110 2024-03-08T08:29:55.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa959a640 1 -- 172.21.15.12:0/1408471552 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607701906 0 0) 0x7fbaac15e1a0 con 0x7fbaac15ddc0 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.481+0000 7fbaa9d9b640 1 --2- 172.21.15.12:0/1408471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d1110 0x7fbaac0b6be0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00097a0 tx=0x7fbaa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0616fa6b87939b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbaa959a640 1 -- 172.21.15.12:0/1408471552 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0035020 con 0x7fbaac0d1110 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbaa959a640 1 -- 172.21.15.12:0/1408471552 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa0031830 con 0x7fbaac0d1110 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbaa959a640 1 -- 172.21.15.12:0/1408471552 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0031c00 con 0x7fbaac0d1110 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 -- 172.21.15.12:0/1408471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d1110 msgr2=0x7fbaac0b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/1408471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d1110 0x7fbaac0b6be0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00097a0 tx=0x7fbaa002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 -- 172.21.15.12:0/1408471552 shutdown_connections 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/1408471552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d31c0 0x7fbaac0b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/1408471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d1110 0x7fbaac0b6be0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 -- 172.21.15.12:0/1408471552 >> 172.21.15.12:0/1408471552 conn(0x7fbaac072720 msgr2=0x7fbaac072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 -- 172.21.15.12:0/1408471552 shutdown_connections 2024-03-08T08:29:55.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.482+0000 7fbab0d9a640 1 -- 172.21.15.12:0/1408471552 wait complete. 2024-03-08T08:29:55.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 Processor -- start 2024-03-08T08:29:55.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 -- start start 2024-03-08T08:29:55.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 0x7fbaac156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 0x7fbaac149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac0b9980 con 0x7fbaac0d31c0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac0b8d50 con 0x7fbaac15ddc0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbab0d9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac0b7660 con 0x7fbaac0d1110 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbaa9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbaa9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52078/0 (socket says 172.21.15.12:52078) 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.484+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/4020585111 learned_addr learned my addr 172.21.15.12:0/4020585111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaaa59c640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 0x7fbaac156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaaad9d640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 0x7fbaac149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 msgr2=0x7fbaac156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 0x7fbaac156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 msgr2=0x7fbaac149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 0x7fbaac149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 -- 172.21.15.12:0/4020585111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac1540d0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaaa59c640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 0x7fbaac156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaaad9d640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 0x7fbaac149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbaa9d9b640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00098c0 tx=0x7fbaa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0004230 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa00043d0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaac14a2d0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa003cad0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.485+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaac1569e0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.486+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaac15e750 con 0x7fbaac0d31c0 2024-03-08T08:29:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.487+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaa0038020 con 0x7fbaac0d31c0 2024-03-08T08:29:55.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.487+0000 7fba9affd640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 0x7fba8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.487+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbaa00ac430 con 0x7fbaac0d31c0 2024-03-08T08:29:55.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.487+0000 7fbaaa59c640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 0x7fba8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.487+0000 7fbaaa59c640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 0x7fba8c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4000f60 tx=0x7fbaa4007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.491+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbaa0041070 con 0x7fbaac0d31c0 2024-03-08T08:29:55.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.709+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbaac0749c0 con 0x7fbaac0d31c0 2024-03-08T08:29:55.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-08T08:29:55.709+0000 7f49b9ba2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T08:29:55.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.711+0000 7fba9affd640 1 -- 172.21.15.12:0/4020585111 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbaa007b440 con 0x7fbaac0d31c0 2024-03-08T08:29:55.712 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-08T08:29:55.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 msgr2=0x7fba8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 0x7fba8c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4000f60 tx=0x7fbaa4007ca0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 msgr2=0x7fbaac153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00098c0 tx=0x7fbaa00046c0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 shutdown_connections 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fba8c06b4d0 0x7fba8c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac15ddc0 0x7fbaac149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaac0d31c0 0x7fbaac153b90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 --2- 172.21.15.12:0/4020585111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbaac0d1110 0x7fbaac156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 >> 172.21.15.12:0/4020585111 conn(0x7fbaac072720 msgr2=0x7fbaac06c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 shutdown_connections 2024-03-08T08:29:55.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.714+0000 7fbab0d9a640 1 -- 172.21.15.12:0/4020585111 wait complete. 2024-03-08T08:29:55.716 INFO:tasks.ceph.mds.a.smithi012.stderr: -3033> 2024-03-08T08:29:10.286+0000 7f0963ea2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-08T08:29:55.726 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-08T08:29:55.743 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-08T08:29:55.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.828+0000 7f94fd9e9640 1 Processor -- start 2024-03-08T08:29:55.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.828+0000 7f94fd9e9640 1 -- start start 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.829+0000 7f94fd9e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.829+0000 7f94fd9e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.829+0000 7f94fd9e9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f815e270 con 0x7f94f815f270 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.829+0000 7f94fd9e9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f80703d0 con 0x7f94f80cd5c0 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.829+0000 7f94fd9e9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f8070610 con 0x7f94f806abd0 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f6ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:53072/0 (socket says 172.21.15.12:53072) 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f77fe640 1 -- 172.21.15.12:0/1697509376 learned_addr learned my addr 172.21.15.12:0/1697509376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f67fc640 1 --1- 172.21.15.12:0/1697509376 >> v1:172.21.15.156:6789/0 conn(0x7f94f815f270 0x7f94f806fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:52734/0 (socket says 172.21.15.12:52734) 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f6ffd640 1 -- 172.21.15.12:0/1697509376 >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 msgr2=0x7f94f80d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f6ffd640 1 --2- 172.21.15.12:0/1697509376 >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f6ffd640 1 -- 172.21.15.12:0/1697509376 >> v1:172.21.15.156:6789/0 conn(0x7f94f815f270 legacy=0x7f94f806fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f6ffd640 1 -- 172.21.15.12:0/1697509376 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f815e4f0 con 0x7f94f806abd0 2024-03-08T08:29:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.830+0000 7f94f77fe640 1 --2- 172.21.15.12:0/1697509376 >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94f6ffd640 1 --2- 172.21.15.12:0/1697509376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f806afb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94f00097a0 tx=0x7f94f002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4572389805fb71c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94f5ffb640 1 -- 172.21.15.12:0/1697509376 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0035020 con 0x7f94f806abd0 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94f5ffb640 1 -- 172.21.15.12:0/1697509376 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f0031830 con 0x7f94f806abd0 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94f5ffb640 1 -- 172.21.15.12:0/1697509376 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0031c00 con 0x7f94f806abd0 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1697509376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 msgr2=0x7f94f806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1697509376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f806afb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94f00097a0 tx=0x7f94f002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1697509376 shutdown_connections 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1697509376 >> v2:172.21.15.156:3300/0 conn(0x7f94f80cd5c0 0x7f94f80d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.831+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1697509376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f806afb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.832+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1697509376 >> 172.21.15.12:0/1697509376 conn(0x7f94f8099a40 msgr2=0x7f94f8099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.832+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1697509376 shutdown_connections 2024-03-08T08:29:55.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.832+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1697509376 wait complete. 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 Processor -- start 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 -- start start 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f809e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 0x7f94f80b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f815ea70 con 0x7f94f815f270 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f815e790 con 0x7f94f80cd5c0 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.833+0000 7f94fd9e9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f815ed50 con 0x7f94f806abd0 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52112/0 (socket says 172.21.15.12:52112) 2024-03-08T08:29:55.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 -- 172.21.15.12:0/1486772837 learned_addr learned my addr 172.21.15.12:0/1486772837 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f67fc640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 0x7f94f80b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f6ffd640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f809e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 -- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 msgr2=0x7f94f809e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f809e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 -- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 msgr2=0x7f94f80b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 0x7f94f80b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 -- 172.21.15.12:0/1486772837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f809e7a0 con 0x7f94f815f270 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f67fc640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 0x7f94f80b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94f77fe640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f94e800efc0 tx=0x7f94e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e800cbe0 con 0x7f94f815f270 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e800cd80 con 0x7f94f815f270 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f80b4990 con 0x7f94f815f270 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8003bb0 con 0x7f94f815f270 2024-03-08T08:29:55.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.834+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f80b4f50 con 0x7f94f815f270 2024-03-08T08:29:55.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.835+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f80749c0 con 0x7f94f815f270 2024-03-08T08:29:55.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.836+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94e801b030 con 0x7f94f815f270 2024-03-08T08:29:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.836+0000 7f94effff640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 0x7f94d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.836+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f94e808a700 con 0x7f94f815f270 2024-03-08T08:29:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.836+0000 7f94f6ffd640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 0x7f94d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.837+0000 7f94f6ffd640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 0x7f94d806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f94f0036000 tx=0x7f94f0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:55.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:55.841+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f94e8022020 con 0x7f94f815f270 2024-03-08T08:29:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.057+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f94f80cd3a0 con 0x7f94f815f270 2024-03-08T08:29:56.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.712+0000 7f94effff640 1 -- 172.21.15.12:0/1486772837 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f94f809e7a0 con 0x7f94f815f270 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.714+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 msgr2=0x7f94d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.714+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 0x7f94d806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f94f0036000 tx=0x7f94f0008030 comp rx=0 tx=0).stop 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 msgr2=0x7f94f80c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f94e800efc0 tx=0x7f94e800c490 comp rx=0 tx=0).stop 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 shutdown_connections 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94d806b4d0 0x7f94d806d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f815f270 0x7f94f80c5680 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f80cd5c0 0x7f94f80b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 --2- 172.21.15.12:0/1486772837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f806abd0 0x7f94f809e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 >> 172.21.15.12:0/1486772837 conn(0x7f94f8099a40 msgr2=0x7f94f80cc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 shutdown_connections 2024-03-08T08:29:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.715+0000 7f94fd9e9640 1 -- 172.21.15.12:0/1486772837 wait complete. 2024-03-08T08:29:56.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-08T08:29:56.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.881+0000 7fc692c47640 1 Processor -- start 2024-03-08T08:29:56.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.881+0000 7fc692c47640 1 -- start start 2024-03-08T08:29:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc692c47640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c15db70 0x7fc68c099c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc692c47640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68c06bce0 0x7fc68c09a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc692c47640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c06c230 con 0x7fc68c1617d0 2024-03-08T08:29:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc692c47640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68c06ce40 con 0x7fc68c06bce0 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc692c47640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c06c510 con 0x7fc68c15db70 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68c06bce0 0x7fc68c09a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68c06bce0 0x7fc68c09a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52126/0 (socket says 172.21.15.12:52126) 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 -- 172.21.15.12:0/956679257 learned_addr learned my addr 172.21.15.12:0/956679257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc6909e3640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c15db70 0x7fc68c099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 -- 172.21.15.12:0/956679257 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c15db70 msgr2=0x7fc68c099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c15db70 0x7fc68c099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 -- 172.21.15.12:0/956679257 >> v1:172.21.15.156:6789/0 conn(0x7fc68c1617d0 legacy=0x7fc68c161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.882+0000 7fc68bfff640 1 -- 172.21.15.12:0/956679257 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c06c7f0 con 0x7fc68c06bce0 2024-03-08T08:29:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.883+0000 7fc68bfff640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c06bce0 0x7fc68c09a140 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc678009870 tx=0x7fc67802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d043bb7edc8ba31 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.883+0000 7fc68b7fe640 1 -- 172.21.15.12:0/956679257 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678034070 con 0x7fc68c06bce0 2024-03-08T08:29:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.883+0000 7fc68b7fe640 1 -- 172.21.15.12:0/956679257 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc678030840 con 0x7fc68c06bce0 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.883+0000 7fc68b7fe640 1 -- 172.21.15.12:0/956679257 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678030c10 con 0x7fc68c06bce0 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 -- 172.21.15.12:0/956679257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c06bce0 msgr2=0x7fc68c09a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c06bce0 0x7fc68c09a140 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc678009870 tx=0x7fc67802b980 comp rx=0 tx=0).stop 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 -- 172.21.15.12:0/956679257 shutdown_connections 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c06bce0 0x7fc68c09a140 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 --2- 172.21.15.12:0/956679257 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c15db70 0x7fc68c099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 -- 172.21.15.12:0/956679257 >> 172.21.15.12:0/956679257 conn(0x7fc68c0b3430 msgr2=0x7fc68c0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 -- 172.21.15.12:0/956679257 shutdown_connections 2024-03-08T08:29:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.884+0000 7fc692c47640 1 -- 172.21.15.12:0/956679257 wait complete. 2024-03-08T08:29:56.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.885+0000 7fc692c47640 1 Processor -- start 2024-03-08T08:29:56.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.885+0000 7fc692c47640 1 -- start start 2024-03-08T08:29:56.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c06bce0 0x7fc68c152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 0x7fc68c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c15e0d0 con 0x7fc68c1617d0 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c15df50 con 0x7fc68c15db70 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc692c47640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c162370 con 0x7fc68c06bce0 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52138/0 (socket says 172.21.15.12:52138) 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 -- 172.21.15.12:0/522085365 learned_addr learned my addr 172.21.15.12:0/522085365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc68bfff640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 0x7fc68c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6909e3640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c06bce0 0x7fc68c152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 -- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c06bce0 msgr2=0x7fc68c152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c06bce0 0x7fc68c152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 -- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 msgr2=0x7fc68c09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 0x7fc68c09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:56.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 -- 172.21.15.12:0/522085365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c0b6920 con 0x7fc68c1617d0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc68bfff640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 0x7fc68c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6911e4640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc68400c760 tx=0x7fc68400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.886+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684016020 con 0x7fc68c1617d0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.887+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc684006d50 con 0x7fc68c1617d0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.887+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c1f7980 con 0x7fc68c1617d0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.887+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684003bb0 con 0x7fc68c1617d0 2024-03-08T08:29:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.887+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c1f7f40 con 0x7fc68c1617d0 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.888+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c0749c0 con 0x7fc68c1617d0 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.888+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6840050d0 con 0x7fc68c1617d0 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.889+0000 7fc6897fa640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.889+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fc684089dd0 con 0x7fc68c1617d0 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.889+0000 7fc6909e3640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.889+0000 7fc6909e3640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 0x7fc67006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc680009ad0 tx=0x7fc680008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:56.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:56.893+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc684020020 con 0x7fc68c1617d0 2024-03-08T08:29:57.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.074+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc68c0c81c0 con 0x7fc68c1617d0 2024-03-08T08:29:57.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.716+0000 7fc6897fa640 1 -- 172.21.15.12:0/522085365 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc684058c70 con 0x7fc68c1617d0 2024-03-08T08:29:57.717 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 msgr2=0x7fc67006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 0x7fc67006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc680009ad0 tx=0x7fc680008040 comp rx=0 tx=0).stop 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 msgr2=0x7fc68c09f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc68400c760 tx=0x7fc68400cb60 comp rx=0 tx=0).stop 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 shutdown_connections 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68c1617d0 0x7fc68c09f020 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc68c15db70 0x7fc68c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 --2- 172.21.15.12:0/522085365 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc68c06bce0 0x7fc68c152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 >> 172.21.15.12:0/522085365 conn(0x7fc68c0b3430 msgr2=0x7fc68c099c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 shutdown_connections 2024-03-08T08:29:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.719+0000 7fc692c47640 1 -- 172.21.15.12:0/522085365 wait complete. 2024-03-08T08:29:57.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-08T08:29:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.845+0000 7f9d09d81640 1 Processor -- start 2024-03-08T08:29:57.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.845+0000 7f9d09d81640 1 -- start start 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d09d81640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9d0415db10 0x7f9d040cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d09d81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d040ccc10 0x7f9d04099c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d09d81640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040cd1b0 con 0x7f9d0409a140 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d09d81640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040cdd30 con 0x7f9d0415db10 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d09d81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040cd490 con 0x7f9d040ccc10 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d037fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9d0415db10 0x7f9d040cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d03fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9d0409a140 0x7f9d04161800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56456/0 (socket says 172.21.15.12:56456) 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d037fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9d0415db10 0x7f9d040cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52140/0 (socket says 172.21.15.12:52140) 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.846+0000 7f9d03fff640 1 -- 172.21.15.12:0/3841327419 learned_addr learned my addr 172.21.15.12:0/3841327419 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d037fe640 1 -- 172.21.15.12:0/3841327419 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d040ccc10 msgr2=0x7f9d04099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d037fe640 1 --2- 172.21.15.12:0/3841327419 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d040ccc10 0x7f9d04099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d037fe640 1 -- 172.21.15.12:0/3841327419 >> v1:172.21.15.12:6789/0 conn(0x7f9d0409a140 legacy=0x7f9d04161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d037fe640 1 -- 172.21.15.12:0/3841327419 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040cd770 con 0x7f9d0415db10 2024-03-08T08:29:57.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d027fc640 1 -- 172.21.15.12:0/3841327419 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753854357 0 0) 0x7f9d040cd1b0 con 0x7f9d0409a140 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.847+0000 7f9d037fe640 1 --2- 172.21.15.12:0/3841327419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0415db10 0x7f9d040cc6d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c174022848d622 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d027fc640 1 -- 172.21.15.12:0/3841327419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4035020 con 0x7f9d0415db10 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d027fc640 1 -- 172.21.15.12:0/3841327419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4031830 con 0x7f9d0415db10 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d027fc640 1 -- 172.21.15.12:0/3841327419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4031c00 con 0x7f9d0415db10 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 -- 172.21.15.12:0/3841327419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0415db10 msgr2=0x7f9d040cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 --2- 172.21.15.12:0/3841327419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0415db10 0x7f9d040cc6d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 -- 172.21.15.12:0/3841327419 shutdown_connections 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 --2- 172.21.15.12:0/3841327419 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d040ccc10 0x7f9d04099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 --2- 172.21.15.12:0/3841327419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0415db10 0x7f9d040cc6d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 -- 172.21.15.12:0/3841327419 >> 172.21.15.12:0/3841327419 conn(0x7f9d040b33d0 msgr2=0x7f9d040b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:57.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 -- 172.21.15.12:0/3841327419 shutdown_connections 2024-03-08T08:29:57.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.848+0000 7f9d09d81640 1 -- 172.21.15.12:0/3841327419 wait complete. 2024-03-08T08:29:57.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.849+0000 7f9d09d81640 1 Processor -- start 2024-03-08T08:29:57.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.849+0000 7f9d09d81640 1 -- start start 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0409a140 0x7f9d040cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d0415db10 0x7f9d040b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d0bf0 con 0x7f9d040ccc10 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d0a70 con 0x7f9d0409a140 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d09d81640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0415def0 con 0x7f9d0415db10 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d02ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d02ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52154/0 (socket says 172.21.15.12:52154) 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d02ffd640 1 -- 172.21.15.12:0/1146879068 learned_addr learned my addr 172.21.15.12:0/1146879068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d037fe640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0409a140 0x7f9d040cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.850+0000 7f9d03fff640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d0415db10 0x7f9d040b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 -- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d0415db10 msgr2=0x7f9d040b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d0415db10 0x7f9d040b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 -- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0409a140 msgr2=0x7f9d040cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0409a140 0x7f9d040cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 -- 172.21.15.12:0/1146879068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040b7760 con 0x7f9d040ccc10 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d02ffd640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800b4d0 tx=0x7f9cf800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8014070 con 0x7f9d040ccc10 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8010870 con 0x7f9d040ccc10 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d040b6700 con 0x7f9d040ccc10 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8010c60 con 0x7f9d040ccc10 2024-03-08T08:29:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.851+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d040b6c80 con 0x7f9d040ccc10 2024-03-08T08:29:57.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.852+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d04068120 con 0x7f9d040ccc10 2024-03-08T08:29:57.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.852+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf8017070 con 0x7f9d040ccc10 2024-03-08T08:29:57.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.853+0000 7f9cebfff640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:29:57.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.853+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f9cf8089ee0 con 0x7f9d040ccc10 2024-03-08T08:29:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.853+0000 7f9d037fe640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:29:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.853+0000 7f9d037fe640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 0x7f9cf006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:29:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:57.857+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9d040b7760 con 0x7f9d040ccc10 2024-03-08T08:29:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.037+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d040749a0 con 0x7f9d040ccc10 2024-03-08T08:29:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.720+0000 7f9cebfff640 1 -- 172.21.15.12:0/1146879068 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9cf8058ea0 con 0x7f9d040ccc10 2024-03-08T08:29:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-08T08:29:58.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 msgr2=0x7f9cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:58.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 0x7f9cf006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf4008030 comp rx=0 tx=0).stop 2024-03-08T08:29:58.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 msgr2=0x7f9d040c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800b4d0 tx=0x7f9cf800b990 comp rx=0 tx=0).stop 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 shutdown_connections 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9d0415db10 0x7f9d040b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d040ccc10 0x7f9d040c9010 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 --2- 172.21.15.12:0/1146879068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0409a140 0x7f9d040cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.723+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 >> 172.21.15.12:0/1146879068 conn(0x7f9d040b33d0 msgr2=0x7f9d0406faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:29:58.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.724+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 shutdown_connections 2024-03-08T08:29:58.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:29:58.724+0000 7f9d09d81640 1 -- 172.21.15.12:0/1146879068 wait complete. 2024-03-08T08:29:58.741 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-08T08:29:58.741 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-08T08:29:58.741 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-08T08:29:58.741 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:58.741 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-08T08:29:58.741 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-08T08:29:58.742 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-08T08:29:58.742 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:58.742 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-08T08:29:58.742 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-08T08:29:58.742 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-08T08:29:58.742 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:58.742 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-08T08:29:58.742 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-08T08:29:58.743 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-08T08:29:58.743 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:29:58.753 INFO:tasks.ceph.mds.c:Stopped 2024-03-08T08:29:58.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-08T08:29:58.754 INFO:tasks.ceph.mds.b:Stopped 2024-03-08T08:29:58.754 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-08T08:29:58.755 INFO:tasks.ceph.mds.d:Stopped 2024-03-08T08:29:58.755 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-08T08:29:58.795 INFO:tasks.ceph.mds.c:Started 2024-03-08T08:29:58.795 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:58.797 INFO:tasks.ceph.mds.b:Started 2024-03-08T08:29:58.797 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:58.798 INFO:tasks.ceph.mds.d:Started 2024-03-08T08:29:58.798 DEBUG:teuthology.parallel:result is None 2024-03-08T08:29:58.877 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-08T08:29:58.882 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2024-03-08T08:29:58.882 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2024-03-08T08:30:02.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-08T08:30:04.753 INFO:tasks.ceph.mds.a:Stopped 2024-03-08T08:30:04.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-08T08:30:04.756 INFO:tasks.ceph.mds.a:Started 2024-03-08T08:30:04.756 DEBUG:teuthology.parallel:result is None 2024-03-08T08:30:04.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-08T08:30:04.843 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-08T08:30:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.887+0000 7fcb9291e640 1 Processor -- start 2024-03-08T08:30:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.887+0000 7fcb9291e640 1 -- start start 2024-03-08T08:30:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb9291e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcb8c0b8300 0x7fcb8c1688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb9291e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c15e1a0 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb9291e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c0b8790 con 0x7fcb8c0d1110 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb9291e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b9030 con 0x7fcb8c0b8300 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcb8c15ddc0 0x7fcb8c0d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49246/0 (socket says 172.21.15.12:49246) 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8bfff640 1 -- 172.21.15.12:0/580347575 learned_addr learned my addr 172.21.15.12:0/580347575 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb90ebb640 1 --2- 172.21.15.12:0/580347575 >> v2:172.21.15.156:3300/0 conn(0x7fcb8c0b8300 0x7fcb8c1688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469052745 0 0) 0x7fcb8c15e1a0 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb74003520 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222590199 0 0) 0x7fcb74003520 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 >> v2:172.21.15.156:3300/0 conn(0x7fcb8c0b8300 msgr2=0x7fcb8c1688d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.888+0000 7fcb8affd640 1 --2- 172.21.15.12:0/580347575 >> v2:172.21.15.156:3300/0 conn(0x7fcb8c0b8300 0x7fcb8c1688d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 >> v1:172.21.15.156:6789/0 conn(0x7fcb8c0d1110 legacy=0x7fcb8c0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0b8a70 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fcb7c001670 con 0x7fcb8c15ddc0 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 >> v1:172.21.15.12:6789/0 conn(0x7fcb8c15ddc0 legacy=0x7fcb8c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb740043b0 0x7fcb74004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb74005080 0x7fcb7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7400bfb0 con 0x7fcb740039b0 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7400c160 con 0x7fcb74005080 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7400c360 con 0x7fcb740043b0 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb90ebb640 1 --2- 172.21.15.12:0/580347575 >> v2:172.21.15.156:3300/0 conn(0x7fcb8c0b8300 0x7fcb8c1688d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8b7fe640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb740043b0 0x7fcb74004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb90ebb640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb74005080 0x7fcb7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 -- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb740043b0 msgr2=0x7fcb74004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb740043b0 0x7fcb74004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 -- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb74005080 msgr2=0x7fcb7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb74005080 0x7fcb7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb90ebb640 1 -- 172.21.15.12:0/580347575 reap_dead start 2024-03-08T08:30:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.889+0000 7fcb8bfff640 1 -- 172.21.15.12:0/580347575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74003520 con 0x7fcb740039b0 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.890+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb74003d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c240 tx=0x7fcb74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea60867f24e6666 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.890+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007d30 con 0x7fcb740039b0 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.890+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c030070 con 0x7fcb740039b0 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.890+0000 7fcb8affd640 1 -- 172.21.15.12:0/580347575 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c005d70 con 0x7fcb740039b0 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 -- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 msgr2=0x7fcb74003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb74003d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c240 tx=0x7fcb74005830 comp rx=0 tx=0).stop 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 -- 172.21.15.12:0/580347575 shutdown_connections 2024-03-08T08:30:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 --2- 172.21.15.12:0/580347575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb74003d90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 -- 172.21.15.12:0/580347575 >> 172.21.15.12:0/580347575 conn(0x7fcb8c072720 msgr2=0x7fcb8c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 -- 172.21.15.12:0/580347575 shutdown_connections 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.891+0000 7fcb9291e640 1 -- 172.21.15.12:0/580347575 wait complete. 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.892+0000 7fcb9291e640 1 Processor -- start 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.892+0000 7fcb9291e640 1 -- start start 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb8c14e780 0x7fcb8c0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8c14ec00 0x7fcb8c0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b9980 con 0x7fcb740039b0 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0d1d20 con 0x7fcb8c14ec00 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb9291e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0d0a00 con 0x7fcb8c14e780 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37958/0 (socket says 172.21.15.12:37958) 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 -- 172.21.15.12:0/3944772831 learned_addr learned my addr 172.21.15.12:0/3944772831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 -- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb8c14e780 msgr2=0x7fcb8c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb8c14e780 0x7fcb8c0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 -- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8c14ec00 msgr2=0x7fcb8c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8c14ec00 0x7fcb8c0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.893+0000 7fcb8bfff640 1 -- 172.21.15.12:0/3944772831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c1f31d0 con 0x7fcb740039b0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb8bfff640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0016a0 tx=0x7fcb7c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007d30 con 0x7fcb740039b0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c004bf0 con 0x7fcb740039b0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c14f160 con 0x7fcb740039b0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007070 con 0x7fcb740039b0 2024-03-08T08:30:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.894+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c14c890 con 0x7fcb740039b0 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.895+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c14f680 con 0x7fcb740039b0 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.895+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb7c048020 con 0x7fcb740039b0 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.896+0000 7fcb88ff9640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 0x7fcb7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.896+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcb7c0ab390 con 0x7fcb740039b0 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.896+0000 7fcb8b7fe640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 0x7fcb7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.896+0000 7fcb8b7fe640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 0x7fcb7806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0b2770 tx=0x7fcb80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:04.900+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcb8c14f680 con 0x7fcb740039b0 2024-03-08T08:30:05.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:05.096+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcb8c0749c0 con 0x7fcb740039b0 2024-03-08T08:30:06.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.068+0000 7fcb88ff9640 1 -- 172.21.15.12:0/3944772831 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v31) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcb7c034e90 con 0x7fcb740039b0 2024-03-08T08:30:06.069 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-08T08:30:06.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 msgr2=0x7fcb7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 0x7fcb7806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0b2770 tx=0x7fcb80009040 comp rx=0 tx=0).stop 2024-03-08T08:30:06.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 msgr2=0x7fcb8c1f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0016a0 tx=0x7fcb7c02e070 comp rx=0 tx=0).stop 2024-03-08T08:30:06.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 shutdown_connections 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcb7806b4d0 0x7fcb7806d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8c14ec00 0x7fcb8c0cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb8c14e780 0x7fcb8c0ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 --2- 172.21.15.12:0/3944772831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb740039b0 0x7fcb8c1f2c90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.071+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 >> 172.21.15.12:0/3944772831 conn(0x7fcb8c072720 msgr2=0x7fcb8c0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.072+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 shutdown_connections 2024-03-08T08:30:06.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.072+0000 7fcb9291e640 1 -- 172.21.15.12:0/3944772831 wait complete. 2024-03-08T08:30:06.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-08T08:30:06.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.193+0000 7f535673e640 1 Processor -- start 2024-03-08T08:30:06.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.193+0000 7f535673e640 1 -- start start 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f535673e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f53501617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f535673e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f53500b94e0 0x7f53500ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f535673e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53501626f0 con 0x7f53500cec60 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f535673e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5350162130 con 0x7f53500b94e0 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f535673e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5350162410 con 0x7f535015dbd0 2024-03-08T08:30:06.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f5354cdb640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f53500cec60 0x7f53500b5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49252/0 (socket says 172.21.15.12:49252) 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.194+0000 7f5354cdb640 1 -- 172.21.15.12:0/1842265603 learned_addr learned my addr 172.21.15.12:0/1842265603 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487962479 0 0) 0x7f53501626f0 con 0x7f53500cec60 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 --2- 172.21.15.12:0/1842265603 >> v2:172.21.15.156:3300/0 conn(0x7f53500b94e0 0x7f53500ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c003520 con 0x7f53500cec60 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534ffff640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f53501617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3038105237 0 0) 0x7f533c003520 con 0x7f53500cec60 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 msgr2=0x7f53501617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f53501617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 >> v2:172.21.15.156:3300/0 conn(0x7f53500b94e0 msgr2=0x7f53500ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 --2- 172.21.15.12:0/1842265603 >> v2:172.21.15.156:3300/0 conn(0x7f53500b94e0 0x7f53500ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500d1c80 con 0x7f53500cec60 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f53480019f0 con 0x7f53500cec60 2024-03-08T08:30:06.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 >> v1:172.21.15.12:6789/0 conn(0x7f53500cec60 legacy=0x7f53500b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f533c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f533c003f30 0x7f533c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c004d40 0x7f533c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5350162130 con 0x7f533c003f30 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5350162410 con 0x7f533c004d40 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c00b9c0 con 0x7f535015dbd0 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f533c003f30 0x7f533c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f5354cdb640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c004d40 0x7f533c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534ffff640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f533c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 -- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 msgr2=0x7f533c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f533c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 -- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c004d40 msgr2=0x7f533c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.195+0000 7f534f7fe640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c004d40 0x7f533c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534f7fe640 1 -- 172.21.15.12:0/1842265603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c003520 con 0x7f533c003f30 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534ffff640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f535015dbd0 0x7f533c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534f7fe640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f533c003f30 0x7f533c004800 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5338009ac0 tx=0x7f533c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b159d002b96a2b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338039070 con 0x7f533c003f30 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533802c6d0 con 0x7f533c003f30 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.196+0000 7f534effd640 1 -- 172.21.15.12:0/1842265603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338009690 con 0x7f533c003f30 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 -- 172.21.15.12:0/1842265603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f533c003f30 msgr2=0x7f533c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 --2- 172.21.15.12:0/1842265603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f533c003f30 0x7f533c004800 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5338009ac0 tx=0x7f533c005830 comp rx=0 tx=0).stop 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f5354cdb640 1 -- 172.21.15.12:0/1842265603 reap_dead start 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 -- 172.21.15.12:0/1842265603 shutdown_connections 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 -- 172.21.15.12:0/1842265603 >> 172.21.15.12:0/1842265603 conn(0x7f53500d3cc0 msgr2=0x7f53500d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 -- 172.21.15.12:0/1842265603 shutdown_connections 2024-03-08T08:30:06.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.197+0000 7f535673e640 1 -- 172.21.15.12:0/1842265603 wait complete. 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.198+0000 7f535673e640 1 Processor -- start 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.198+0000 7f535673e640 1 -- start start 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 0x7f53500c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 0x7f5350150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500d3490 con 0x7f53500b8370 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535015dfb0 con 0x7f533c003f30 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f535673e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500d3610 con 0x7f53500cb530 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37984/0 (socket says 172.21.15.12:37984) 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 -- 172.21.15.12:0/2126030812 learned_addr learned my addr 172.21.15.12:0/2126030812 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f534f7fe640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 0x7f5350150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f534ffff640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 0x7f53500c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 -- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 msgr2=0x7f5350150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 0x7f5350150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 -- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 msgr2=0x7f53500c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.199+0000 7f5354cdb640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 0x7f53500c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f5354cdb640 1 -- 172.21.15.12:0/2126030812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500cbc80 con 0x7f53500b8370 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f534f7fe640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 0x7f5350150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f534ffff640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 0x7f53500c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f5354cdb640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f535008cc80 tx=0x7f5348007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348019070 con 0x7f53500b8370 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534800ab80 con 0x7f53500b8370 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480133f0 con 0x7f53500b8370 2024-03-08T08:30:06.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53500cbf60 con 0x7f53500b8370 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.200+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53500cc520 con 0x7f53500b8370 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.201+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53500b8d30 con 0x7f53500b8370 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.201+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f534801c070 con 0x7f53500b8370 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.202+0000 7f534cff9640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.202+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5348089a70 con 0x7f53500b8370 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.202+0000 7f534ffff640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.202+0000 7f534ffff640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5340006d40 tx=0x7f5340008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.206+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5348058c40 con 0x7f53500b8370 2024-03-08T08:30:06.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.452+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f53500749c0 con 0x7f53500b8370 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.453+0000 7f534cff9640 1 -- 172.21.15.12:0/2126030812 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f5348058540 con 0x7f53500b8370 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.454 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOplDpdVLhAAo1qwVTByZkCW99JvxNG5fw==", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOplDwOYOhAAZZCYzARU9SC14+aUfg4upA==", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-08T08:30:06.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOplAymcNRAA0wcrVZraXrP1J4XjBgfVqQ==", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-08T08:30:06.456 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAhzOplqRpYBhAAfgXmaCR/NePZNhmJNzSbxQ==", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAkzOplpscuJxAAvVJ1o5YDhHfY4zE6BlFAyg==", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAnzOpl6LCTJRAApue+ew4o4RVwgj56sgHhuw==", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAqzOpl1uikIxAAmk+OecUZ40V+w7kBueFp8g==", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAtzOplwfSjIRAAbrGyWTJRwqcwqktutrXUnA==", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-08T08:30:06.459 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAxzOplnpH8LRAAFcm5yk7qNJEQNUoq9VWymQ==", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQA0zOplRRY5LRAAmOK8521OeEEjxKdzdSylxg==", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQA3zOplpQtHKxAALqvEWOSsbexSBWKTehdREQ==", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQA6zOplwMqQKRAAJNPW34xRBzrJP2qTwCQITQ==", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAhzOplr8ebChAA8n4wUJZpCY0XA2qC5ELxpw==", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAhzOplCj7FExAADQsys35++1vKbNpqu8Sigw==", 2024-03-08T08:30:06.462 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAhzOplJi9bEBAA8Xr6QHVKC5Spv8RLRKCrxw==", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-08T08:30:06.463 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAhzOplcTeHGRAAvXhoVFl4INMTIU09S114jw==", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-08T08:30:06.464 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOpl5ANGEhAA8uNXB8dcZxuOWCpi63p+1A==", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mds", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOplPZUmJxAAQjZj49MzAEcE/ehNCAVxEQ==", 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mds" 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mgr", 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOpl0J4mJxAA4R4kl91sR3W84Uv2hdr5TQ==", 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-osd", 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOplXKUmJxAAcCuglRQ9ODa95gsBMaHbtg==", 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-osd" 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd", 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOpljasmJxAAyy8LJEqCXmsS70iT/X0Klw==", 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOplO7ImJxAAZpysQvkuHF8aDHtT3DQitQ==", 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rgw", 2024-03-08T08:30:06.467 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQBAzOplVbkmJxAAH74y2rcYv+5V5beC6+VVtQ==", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOplSrTyKhAAHqlsN/aGIbJH4WlA/120cA==", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-08T08:30:06.468 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQAgzOplSUhfJxAAywlcJde+D6Ms0WSzgWojlw==", 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-08T08:30:06.469 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 msgr2=0x7f533006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5340006d40 tx=0x7f5340008040 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 msgr2=0x7f53500b87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f535008cc80 tx=0x7f5348007b20 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 shutdown_connections 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53500b8370 0x7f53500b87f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53500cb530 0x7f5350150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 --2- 172.21.15.12:0/2126030812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f533c003f30 0x7f53500c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 >> 172.21.15.12:0/2126030812 conn(0x7f53500d3cc0 msgr2=0x7f535015d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 shutdown_connections 2024-03-08T08:30:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.455+0000 7f535673e640 1 -- 172.21.15.12:0/2126030812 wait complete. 2024-03-08T08:30:06.471 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-08T08:30:06.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.573+0000 7f3cdc01c640 1 Processor -- start 2024-03-08T08:30:06.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.573+0000 7f3cdc01c640 1 -- start start 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cdc01c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cd409a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cdc01c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40d0150 con 0x7f3cd406cab0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cdc01c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40d0410 con 0x7f3cd40cfbb0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cdc01c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40d0d40 con 0x7f3cd415dda0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cda5b9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3cd406cab0 0x7f3cd40cbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49260/0 (socket says 172.21.15.12:49260) 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1390287416 learned_addr learned my addr 172.21.15.12:0/1390287416 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd9db8640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cd409a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306702957 0 0) 0x7f3cd40d0150 con 0x7f3cd406cab0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc003520 con 0x7f3cd406cab0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262651422 0 0) 0x7f3cbc003520 con 0x7f3cd406cab0 2024-03-08T08:30:06.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 msgr2=0x7f3cd409a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cd409a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 >> v1:172.21.15.156:6789/0 conn(0x7f3cd40cfbb0 legacy=0x7f3cd406c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.574+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40d06f0 con 0x7f3cd406cab0 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3ccc001670 con 0x7f3cd406cab0 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 >> v1:172.21.15.12:6789/0 conn(0x7f3cd406cab0 legacy=0x7f3cd40cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cbc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cbc003f20 0x7f3cbc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc004d40 0x7f3cbc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40d0d40 con 0x7f3cbc004d40 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc00ba20 con 0x7f3cbc003f20 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc00bc20 con 0x7f3cd415dda0 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc004d40 0x7f3cbc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd9db8640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cbc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cd95b7640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cbc003f20 0x7f3cbc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 msgr2=0x7f3cbc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cd415dda0 0x7f3cbc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cbc003f20 msgr2=0x7f3cbc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cbc003f20 0x7f3cbc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.575+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1390287416 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc003520 con 0x7f3cbc004d40 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc004d40 0x7f3cbc0051c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c190 tx=0x7f3cbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d792d0a8713c46b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc007cc0 con 0x7f3cbc004d40 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc030070 con 0x7f3cbc004d40 2024-03-08T08:30:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cd8db6640 1 -- 172.21.15.12:0/1390287416 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc005d50 con 0x7f3cbc004d40 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1390287416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc004d40 msgr2=0x7f3cbc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1390287416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc004d40 0x7f3cbc0051c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c190 tx=0x7f3cbc005830 comp rx=0 tx=0).stop 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1390287416 reap_dead start 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1390287416 shutdown_connections 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.576+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1390287416 >> 172.21.15.12:0/1390287416 conn(0x7f3cd40b3460 msgr2=0x7f3cd40b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.577+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1390287416 shutdown_connections 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.577+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1390287416 wait complete. 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 Processor -- start 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 -- start start 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cbc004d40 0x7f3cd40c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cd4148490 0x7f3cd4148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd406bf10 con 0x7f3cd41488b0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415e180 con 0x7f3cd4148490 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.578+0000 7f3cdc01c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd409a5a0 con 0x7f3cbc004d40 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38000/0 (socket says 172.21.15.12:38000) 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1237595602 learned_addr learned my addr 172.21.15.12:0/1237595602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cd9db8640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cbc004d40 0x7f3cd40c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cd95b7640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cd4148490 0x7f3cd4148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cbc004d40 msgr2=0x7f3cd40c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cbc004d40 0x7f3cd40c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cd4148490 msgr2=0x7f3cd4148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cd4148490 0x7f3cd4148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 -- 172.21.15.12:0/1237595602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4161470 con 0x7f3cd41488b0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cda5b9640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc02c430 tx=0x7f3ccc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc007cc0 con 0x7f3cd41488b0 2024-03-08T08:30:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc02cbd0 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd40c68e0 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc001c00 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.579+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd40c6ea0 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.580+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd40749c0 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.581+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ccc007070 con 0x7f3cd41488b0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.581+0000 7f3cd27fc640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 0x7f3cc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.581+0000 7f3cd9db8640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 0x7f3cc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.581+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3ccc0ab0d0 con 0x7f3cd41488b0 2024-03-08T08:30:06.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.582+0000 7f3cd9db8640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 0x7f3cc006d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4014510 tx=0x7f3cc400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.586+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ccc07a210 con 0x7f3cd41488b0 2024-03-08T08:30:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.835+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f3cd40b78f0 con 0x7f3cd41488b0 2024-03-08T08:30:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.837+0000 7f3cd27fc640 1 -- 172.21.15.12:0/1237595602 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3ccc079b10 con 0x7f3cd41488b0 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 msgr2=0x7f3cc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 0x7f3cc006d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4014510 tx=0x7f3cc400b040 comp rx=0 tx=0).stop 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 msgr2=0x7f3cd41492e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc02c430 tx=0x7f3ccc02e070 comp rx=0 tx=0).stop 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 shutdown_connections 2024-03-08T08:30:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3cc006b4d0 0x7f3cc006d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd41488b0 0x7f3cd41492e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cd4148490 0x7f3cd4148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 --2- 172.21.15.12:0/1237595602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3cbc004d40 0x7f3cd40c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 >> 172.21.15.12:0/1237595602 conn(0x7f3cd40b3460 msgr2=0x7f3cd406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 shutdown_connections 2024-03-08T08:30:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.839+0000 7f3cdc01c640 1 -- 172.21.15.12:0/1237595602 wait complete. 2024-03-08T08:30:06.851 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-08T08:30:06.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.952+0000 7f73373d7640 1 Processor -- start 2024-03-08T08:30:06.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.952+0000 7f73373d7640 1 -- start start 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f73373d7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73300d1110 0x7f73300b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f73373d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 0x7f73300b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f73373d7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733015e1a0 con 0x7f733015ddc0 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f73373d7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300b9030 con 0x7f73300d1110 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f73373d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300d1d20 con 0x7f73300d31c0 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f7334972640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73300d1110 0x7f73300b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f7334972640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73300d1110 0x7f73300b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38012/0 (socket says 172.21.15.12:38012) 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f7334972640 1 -- 172.21.15.12:0/2465640603 learned_addr learned my addr 172.21.15.12:0/2465640603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.953+0000 7f7335974640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 0x7f73300b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7334972640 1 -- 172.21.15.12:0/2465640603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 msgr2=0x7f73300b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:06.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7334972640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 0x7f73300b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7334972640 1 -- 172.21.15.12:0/2465640603 >> v1:172.21.15.156:6789/0 conn(0x7f733015ddc0 legacy=0x7f73300d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7334972640 1 -- 172.21.15.12:0/2465640603 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300b8790 con 0x7f73300d1110 2024-03-08T08:30:06.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7335974640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 0x7f73300b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f7334972640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73300d1110 0x7f73300b6be0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7324009870 tx=0x7f732402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e137474463761ece server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.954+0000 7f732ffff640 1 -- 172.21.15.12:0/2465640603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324034070 con 0x7f73300d1110 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f732ffff640 1 -- 172.21.15.12:0/2465640603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324030840 con 0x7f73300d1110 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f732ffff640 1 -- 172.21.15.12:0/2465640603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324030c10 con 0x7f73300d1110 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 -- 172.21.15.12:0/2465640603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73300d1110 msgr2=0x7f73300b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73300d1110 0x7f73300b6be0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7324009870 tx=0x7f732402b980 comp rx=0 tx=0).stop 2024-03-08T08:30:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 -- 172.21.15.12:0/2465640603 shutdown_connections 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d31c0 0x7f73300b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 --2- 172.21.15.12:0/2465640603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73300d1110 0x7f73300b6be0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 -- 172.21.15.12:0/2465640603 >> 172.21.15.12:0/2465640603 conn(0x7f7330072720 msgr2=0x7f7330072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 -- 172.21.15.12:0/2465640603 shutdown_connections 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.955+0000 7f73373d7640 1 -- 172.21.15.12:0/2465640603 wait complete. 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.956+0000 7f73373d7640 1 Processor -- start 2024-03-08T08:30:06.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.956+0000 7f73373d7640 1 -- start start 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 0x7f73300cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73300d31c0 0x7f7330160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300b9980 con 0x7f733015ddc0 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300b8d50 con 0x7f73300d31c0 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f73373d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300b7660 con 0x7f73300d1110 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38016/0 (socket says 172.21.15.12:38016) 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 -- 172.21.15.12:0/4107902537 learned_addr learned my addr 172.21.15.12:0/4107902537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:06.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7334972640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73300d31c0 0x7f7330160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335173640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 0x7f73300cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 -- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 msgr2=0x7f73300cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 0x7f73300cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 -- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73300d31c0 msgr2=0x7f7330160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73300d31c0 0x7f7330160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335974640 1 -- 172.21.15.12:0/4107902537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73301611f0 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.957+0000 7f7335173640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 0x7f73300cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f7335974640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7320009fa0 tx=0x7f732000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732001a070 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7320003bb0 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7330153310 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320004030 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.958+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73300cc490 con 0x7f733015ddc0 2024-03-08T08:30:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.959+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73300749c0 con 0x7f733015ddc0 2024-03-08T08:30:06.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.959+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f732001d070 con 0x7f733015ddc0 2024-03-08T08:30:06.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.960+0000 7f732dffb640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 0x7f731c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:06.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.960+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7320089ff0 con 0x7f733015ddc0 2024-03-08T08:30:06.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.960+0000 7f7335173640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 0x7f731c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:06.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.960+0000 7f7335173640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 0x7f731c06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7328001660 tx=0x7f7328009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:06.964+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7330153310 con 0x7f733015ddc0 2024-03-08T08:30:07.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.217+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f73300b3950 con 0x7f733015ddc0 2024-03-08T08:30:07.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.219+0000 7f732dffb640 1 -- 172.21.15.12:0/4107902537 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7320010070 con 0x7f733015ddc0 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 msgr2=0x7f731c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 0x7f731c06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7328001660 tx=0x7f7328009040 comp rx=0 tx=0).stop 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 msgr2=0x7f7330143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7320009fa0 tx=0x7f732000d560 comp rx=0 tx=0).stop 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 shutdown_connections 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f731c06b4d0 0x7f731c06d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f733015ddc0 0x7f7330143f00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73300d31c0 0x7f7330160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 --2- 172.21.15.12:0/4107902537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f73300d1110 0x7f73300cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 >> 172.21.15.12:0/4107902537 conn(0x7f7330072720 msgr2=0x7f73300b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 shutdown_connections 2024-03-08T08:30:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.222+0000 7f73373d7640 1 -- 172.21.15.12:0/4107902537 wait complete. 2024-03-08T08:30:07.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-08T08:30:07.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.335+0000 7f33793d7640 1 Processor -- start 2024-03-08T08:30:07.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.335+0000 7f33793d7640 1 -- start start 2024-03-08T08:30:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33793d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f337406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33793d7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740ce3a0 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33793d7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740ce680 con 0x7f337406e980 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33793d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337406d5e0 con 0x7f3374144e00 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3372ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3374144930 0x7f33740cdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49270/0 (socket says 172.21.15.12:49270) 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3372ffd640 1 -- 172.21.15.12:0/1268753285 learned_addr learned my addr 172.21.15.12:0/1268753285 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33727fc640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f337406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484043029 0 0) 0x7f33740ce3a0 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f335c003520 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803508472 0 0) 0x7f335c003520 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 msgr2=0x7f337406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f337406e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 >> v1:172.21.15.156:6789/0 conn(0x7f337406e980 legacy=0x7f33740cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337406d8c0 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3364001670 con 0x7f3374144930 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 >> v1:172.21.15.12:6789/0 conn(0x7f3374144930 legacy=0x7f33740cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c0039e0 0x7f335c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f335c004800 0x7f335c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f335c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337406d5e0 con 0x7f335c004800 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c00ba70 con 0x7f335c0039e0 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c00bc70 con 0x7f3374144e00 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.336+0000 7f33727fc640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f335c004800 0x7f335c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f3372ffd640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c0039e0 0x7f335c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33737fe640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f335c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 -- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 msgr2=0x7f335c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f335c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 -- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c0039e0 msgr2=0x7f335c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c0039e0 0x7f335c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 -- 172.21.15.12:0/1268753285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c003520 con 0x7f335c004800 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33737fe640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3374144e00 0x7f335c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.337+0000 7f33727fc640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f335c004800 0x7f335c004c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3368009a90 tx=0x7f335c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41db6ba8d185a25b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368039070 con 0x7f335c004800 2024-03-08T08:30:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336802c660 con 0x7f335c004800 2024-03-08T08:30:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f3371ffb640 1 -- 172.21.15.12:0/1268753285 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368009690 con 0x7f335c004800 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 -- 172.21.15.12:0/1268753285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f335c004800 msgr2=0x7f335c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 --2- 172.21.15.12:0/1268753285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f335c004800 0x7f335c004c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3368009a90 tx=0x7f335c005830 comp rx=0 tx=0).stop 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33737fe640 1 -- 172.21.15.12:0/1268753285 reap_dead start 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 -- 172.21.15.12:0/1268753285 shutdown_connections 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 -- 172.21.15.12:0/1268753285 >> 172.21.15.12:0/1268753285 conn(0x7f337406abd0 msgr2=0x7f337406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 -- 172.21.15.12:0/1268753285 shutdown_connections 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.338+0000 7f33793d7640 1 -- 172.21.15.12:0/1268753285 wait complete. 2024-03-08T08:30:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.339+0000 7f33793d7640 1 Processor -- start 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.339+0000 7f33793d7640 1 -- start start 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c004800 0x7f33740b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33740b3100 0x7f33740b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337406e450 con 0x7f337415cb80 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337406e2d0 con 0x7f335c004800 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33793d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337406ee10 con 0x7f33740b3100 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38038/0 (socket says 172.21.15.12:38038) 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33727fc640 1 -- 172.21.15.12:0/2929923583 learned_addr learned my addr 172.21.15.12:0/2929923583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f3372ffd640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c004800 0x7f33740b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.340+0000 7f33737fe640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33740b3100 0x7f33740b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 -- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33740b3100 msgr2=0x7f33740b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33740b3100 0x7f33740b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 -- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c004800 msgr2=0x7f33740b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c004800 0x7f33740b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 -- 172.21.15.12:0/2929923583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337415a6e0 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33727fc640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3368000c00 tx=0x7f3368004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368042020 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336803a040 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368039070 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337415a9c0 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.341+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337415af80 con 0x7f337415cb80 2024-03-08T08:30:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.342+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33740a84a0 con 0x7f337415cb80 2024-03-08T08:30:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.342+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3368045020 con 0x7f337415cb80 2024-03-08T08:30:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.343+0000 7f335b7fe640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 0x7f336006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.343+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33680ab2c0 con 0x7f337415cb80 2024-03-08T08:30:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.343+0000 7f3372ffd640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 0x7f336006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.343+0000 7f3372ffd640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 0x7f336006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3364001eb0 tx=0x7f3364009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.347+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33740a84a0 con 0x7f337415cb80 2024-03-08T08:30:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.600+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f33740749c0 con 0x7f337415cb80 2024-03-08T08:30:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.602+0000 7f335b7fe640 1 -- 172.21.15.12:0/2929923583 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f336804c070 con 0x7f337415cb80 2024-03-08T08:30:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 msgr2=0x7f336006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 0x7f336006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3364001eb0 tx=0x7f3364009040 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 msgr2=0x7f337415d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3368000c00 tx=0x7f3368004730 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 shutdown_connections 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f336006b500 0x7f336006d9c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33740b3100 0x7f33740b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f337415cb80 0x7f337415d000 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 --2- 172.21.15.12:0/2929923583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f335c004800 0x7f33740b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 >> 172.21.15.12:0/2929923583 conn(0x7f337406abd0 msgr2=0x7f33740af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 shutdown_connections 2024-03-08T08:30:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.604+0000 7f33793d7640 1 -- 172.21.15.12:0/2929923583 wait complete. 2024-03-08T08:30:07.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-08T08:30:07.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.717+0000 7f6d1546f640 1 Processor -- start 2024-03-08T08:30:07.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.717+0000 7f6d1546f640 1 -- start start 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d1015dee0 0x7f6d100b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 0x7f6d1006abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1006b110 0x7f6d100d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d100ad780 con 0x7f6d1015dee0 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d100b38b0 con 0x7f6d100b5510 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d1546f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100b3b90 con 0x7f6d1006b110 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d1015dee0 0x7f6d100b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d1015dee0 0x7f6d100b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38042/0 (socket says 172.21.15.12:38042) 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 -- 172.21.15.12:0/859582559 learned_addr learned my addr 172.21.15.12:0/859582559 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0e7fc640 1 --2- 172.21.15.12:0/859582559 >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 0x7f6d1006abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0f7fe640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1006b110 0x7f6d100d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 -- 172.21.15.12:0/859582559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1006b110 msgr2=0x7f6d100d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1006b110 0x7f6d100d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 -- 172.21.15.12:0/859582559 >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 msgr2=0x7f6d1006abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 --2- 172.21.15.12:0/859582559 >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 0x7f6d1006abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0effd640 1 -- 172.21.15.12:0/859582559 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100b3e70 con 0x7f6d1015dee0 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.718+0000 7f6d0e7fc640 1 --2- 172.21.15.12:0/859582559 >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 0x7f6d1006abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.719+0000 7f6d0effd640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1015dee0 0x7f6d100b1ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d1006bcb0 tx=0x7f6d0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2d59c6a40e2eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.719+0000 7f6d0dffb640 1 -- 172.21.15.12:0/859582559 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08035020 con 0x7f6d1015dee0 2024-03-08T08:30:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.719+0000 7f6d0dffb640 1 -- 172.21.15.12:0/859582559 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d08031870 con 0x7f6d1015dee0 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.719+0000 7f6d0dffb640 1 -- 172.21.15.12:0/859582559 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08031c40 con 0x7f6d1015dee0 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 -- 172.21.15.12:0/859582559 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1015dee0 msgr2=0x7f6d100b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1015dee0 0x7f6d100b1ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d1006bcb0 tx=0x7f6d0802b910 comp rx=0 tx=0).stop 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 -- 172.21.15.12:0/859582559 shutdown_connections 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1006b110 0x7f6d100d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 --2- 172.21.15.12:0/859582559 >> v2:172.21.15.156:3300/0 conn(0x7f6d100b5510 0x7f6d1006abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 --2- 172.21.15.12:0/859582559 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1015dee0 0x7f6d100b1ea0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 -- 172.21.15.12:0/859582559 >> 172.21.15.12:0/859582559 conn(0x7f6d1006c9d0 msgr2=0x7f6d10099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 -- 172.21.15.12:0/859582559 shutdown_connections 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.720+0000 7f6d1546f640 1 -- 172.21.15.12:0/859582559 wait complete. 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.721+0000 7f6d1546f640 1 Processor -- start 2024-03-08T08:30:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.721+0000 7f6d1546f640 1 -- start start 2024-03-08T08:30:07.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d100b5510 0x7f6d1015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1015dee0 0x7f6d10150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100d0a60 con 0x7f6d1006b110 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100b5c70 con 0x7f6d100b5510 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d1546f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100d0be0 con 0x7f6d1015dee0 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38050/0 (socket says 172.21.15.12:38050) 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 -- 172.21.15.12:0/3624547050 learned_addr learned my addr 172.21.15.12:0/3624547050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 -- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1015dee0 msgr2=0x7f6d10150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1015dee0 0x7f6d10150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 -- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d100b5510 msgr2=0x7f6d1015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d100b5510 0x7f6d1015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 -- 172.21.15.12:0/3624547050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1015f7a0 con 0x7f6d1006b110 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.722+0000 7f6d0effd640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d0802b950 tx=0x7f6d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.723+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08035040 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.723+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d08045480 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.723+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1015fa80 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.723+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08045870 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.723+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1014eb10 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.724+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d100749c0 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.724+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d08038020 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.724+0000 7f6d077fe640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.724+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6d080ac2f0 con 0x7f6d1006b110 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.725+0000 7f6d0e7fc640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.725+0000 7f6d0e7fc640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 0x7f6cf006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:07.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.729+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6d0807b550 con 0x7f6d1006b110 2024-03-08T08:30:07.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.979+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f6d100b2f60 con 0x7f6d1006b110 2024-03-08T08:30:07.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.981+0000 7f6d077fe640 1 -- 172.21.15.12:0/3624547050 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6d0807ae50 con 0x7f6d1006b110 2024-03-08T08:30:07.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 msgr2=0x7f6cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 0x7f6cf006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-03-08T08:30:07.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 msgr2=0x7f6d101453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d0802b950 tx=0x7f6d080046c0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 shutdown_connections 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d1015dee0 0x7f6d10150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d100b5510 0x7f6d1015f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 --2- 172.21.15.12:0/3624547050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d1006b110 0x7f6d101453c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 >> 172.21.15.12:0/3624547050 conn(0x7f6d1006c9d0 msgr2=0x7f6d100aec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 shutdown_connections 2024-03-08T08:30:07.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:07.983+0000 7f6d1546f640 1 -- 172.21.15.12:0/3624547050 wait complete. 2024-03-08T08:30:07.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-08T08:30:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.096+0000 7fbf67265640 1 Processor -- start 2024-03-08T08:30:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 -- start start 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf6015dd80 0x7fbf600b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf600b1f80 0x7fbf600b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf600b57b0 0x7fbf600b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600af360 con 0x7fbf600b1f80 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600b0850 con 0x7fbf6015dd80 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.097+0000 7fbf67265640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600d0c70 con 0x7fbf600b57b0 2024-03-08T08:30:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf600b1f80 0x7fbf600b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf600b1f80 0x7fbf600b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38062/0 (socket says 172.21.15.12:38062) 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 -- 172.21.15.12:0/3890209485 learned_addr learned my addr 172.21.15.12:0/3890209485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf65001640 1 --2- 172.21.15.12:0/3890209485 >> v2:172.21.15.156:3300/0 conn(0x7fbf6015dd80 0x7fbf600b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 -- 172.21.15.12:0/3890209485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf600b57b0 msgr2=0x7fbf600b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf65802640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf600b57b0 0x7fbf600b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf600b57b0 0x7fbf600b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 -- 172.21.15.12:0/3890209485 >> v2:172.21.15.156:3300/0 conn(0x7fbf6015dd80 msgr2=0x7fbf600b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 --2- 172.21.15.12:0/3890209485 >> v2:172.21.15.156:3300/0 conn(0x7fbf6015dd80 0x7fbf600b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.098+0000 7fbf64800640 1 -- 172.21.15.12:0/3890209485 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600d0f50 con 0x7fbf600b1f80 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf64800640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf600b5270 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbf60067940 tx=0x7fbf5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c985a3dfa3997f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf57fff640 1 -- 172.21.15.12:0/3890209485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50035020 con 0x7fbf600b1f80 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf57fff640 1 -- 172.21.15.12:0/3890209485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50031870 con 0x7fbf600b1f80 2024-03-08T08:30:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf57fff640 1 -- 172.21.15.12:0/3890209485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50031c40 con 0x7fbf600b1f80 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 -- 172.21.15.12:0/3890209485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 msgr2=0x7fbf600b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf600b5270 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbf60067940 tx=0x7fbf5002b910 comp rx=0 tx=0).stop 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 -- 172.21.15.12:0/3890209485 shutdown_connections 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf600b57b0 0x7fbf600b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 --2- 172.21.15.12:0/3890209485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf600b5270 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 --2- 172.21.15.12:0/3890209485 >> v2:172.21.15.156:3300/0 conn(0x7fbf6015dd80 0x7fbf600b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.099+0000 7fbf67265640 1 -- 172.21.15.12:0/3890209485 >> 172.21.15.12:0/3890209485 conn(0x7fbf60099a40 msgr2=0x7fbf60099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.100+0000 7fbf67265640 1 -- 172.21.15.12:0/3890209485 shutdown_connections 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.100+0000 7fbf67265640 1 -- 172.21.15.12:0/3890209485 wait complete. 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 Processor -- start 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 -- start start 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf600b57b0 0x7fbf601eaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf6015dd80 0x7fbf6009d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b6200 con 0x7fbf600b1f80 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600d1510 con 0x7fbf600b57b0 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf67265640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b6380 con 0x7fbf6015dd80 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.101+0000 7fbf65001640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38078/0 (socket says 172.21.15.12:38078) 2024-03-08T08:30:08.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 -- 172.21.15.12:0/638184887 learned_addr learned my addr 172.21.15.12:0/638184887 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65802640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf6015dd80 0x7fbf6009d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf64800640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf600b57b0 0x7fbf601eaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 -- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf6015dd80 msgr2=0x7fbf6009d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf6015dd80 0x7fbf6009d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 -- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf600b57b0 msgr2=0x7fbf601eaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf600b57b0 0x7fbf601eaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 -- 172.21.15.12:0/638184887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6014afd0 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf65001640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00c7f0 tx=0x7fbf5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c015070 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c004220 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c007620 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6014b310 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.102+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6014bb50 con 0x7fbf600b1f80 2024-03-08T08:30:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.103+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600749c0 con 0x7fbf600b1f80 2024-03-08T08:30:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.104+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf5c0079d0 con 0x7fbf600b1f80 2024-03-08T08:30:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.104+0000 7fbf55ffb640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 0x7fbf4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.104+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf5c08a110 con 0x7fbf600b1f80 2024-03-08T08:30:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.104+0000 7fbf64800640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 0x7fbf4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.105+0000 7fbf64800640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 0x7fbf4406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf50036000 tx=0x7fbf5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.108+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbf6014b310 con 0x7fbf600b1f80 2024-03-08T08:30:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.358+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fbf6014b4f0 con 0x7fbf600b1f80 2024-03-08T08:30:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.360+0000 7fbf55ffb640 1 -- 172.21.15.12:0/638184887 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbf5c058c90 con 0x7fbf600b1f80 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 msgr2=0x7fbf4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 0x7fbf4406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf50036000 tx=0x7fbf5003c040 comp rx=0 tx=0).stop 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 msgr2=0x7fbf601e68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00c7f0 tx=0x7fbf5c00ccb0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 shutdown_connections 2024-03-08T08:30:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbf4406b4d0 0x7fbf4406d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf6015dd80 0x7fbf6009d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf600b57b0 0x7fbf601eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 --2- 172.21.15.12:0/638184887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf600b1f80 0x7fbf601e68b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 >> 172.21.15.12:0/638184887 conn(0x7fbf60099a40 msgr2=0x7fbf600adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 shutdown_connections 2024-03-08T08:30:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.363+0000 7fbf67265640 1 -- 172.21.15.12:0/638184887 wait complete. 2024-03-08T08:30:08.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-08T08:30:08.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.476+0000 7f765730d640 1 Processor -- start 2024-03-08T08:30:08.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.476+0000 7f765730d640 1 -- start start 2024-03-08T08:30:08.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f765730d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f76500d1110 0x7f76500b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f765730d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 0x7f76500b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f765730d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765015e1a0 con 0x7f765015ddc0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f765730d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76500b9030 con 0x7f76500d1110 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f765730d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76500d1d20 con 0x7f76500d31c0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76550a9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f765015ddc0 0x7f76500d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48122/0 (socket says 172.21.15.12:48122) 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76548a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f76500d1110 0x7f76500b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76550a9640 1 -- 172.21.15.12:0/1413667608 learned_addr learned my addr 172.21.15.12:0/1413667608 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76558aa640 1 --2- 172.21.15.12:0/1413667608 >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 0x7f76500b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f7647fff640 1 -- 172.21.15.12:0/1413667608 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265806279 0 0) 0x7f765015e1a0 con 0x7f765015ddc0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f7647fff640 1 -- 172.21.15.12:0/1413667608 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f763c003520 con 0x7f765015ddc0 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76548a8640 1 -- 172.21.15.12:0/1413667608 >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 msgr2=0x7f76500b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76548a8640 1 --2- 172.21.15.12:0/1413667608 >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 0x7f76500b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76548a8640 1 -- 172.21.15.12:0/1413667608 >> v1:172.21.15.12:6789/0 conn(0x7f765015ddc0 legacy=0x7f76500d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76548a8640 1 -- 172.21.15.12:0/1413667608 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500b8790 con 0x7f76500d1110 2024-03-08T08:30:08.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.477+0000 7f76558aa640 1 --2- 172.21.15.12:0/1413667608 >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 0x7f76500b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.478+0000 7f76548a8640 1 --2- 172.21.15.12:0/1413667608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76500d1110 0x7f76500b6be0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f76500ce090 tx=0x7f764002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4094fa0ce732d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.478+0000 7f7647fff640 1 -- 172.21.15.12:0/1413667608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640035020 con 0x7f76500d1110 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.478+0000 7f7647fff640 1 -- 172.21.15.12:0/1413667608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7640031830 con 0x7f76500d1110 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.478+0000 7f7647fff640 1 -- 172.21.15.12:0/1413667608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640031c00 con 0x7f76500d1110 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 -- 172.21.15.12:0/1413667608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76500d1110 msgr2=0x7f76500b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 --2- 172.21.15.12:0/1413667608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76500d1110 0x7f76500b6be0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f76500ce090 tx=0x7f764002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 -- 172.21.15.12:0/1413667608 shutdown_connections 2024-03-08T08:30:08.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 --2- 172.21.15.12:0/1413667608 >> v2:172.21.15.156:3300/0 conn(0x7f76500d31c0 0x7f76500b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 --2- 172.21.15.12:0/1413667608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76500d1110 0x7f76500b6be0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 -- 172.21.15.12:0/1413667608 >> 172.21.15.12:0/1413667608 conn(0x7f7650072720 msgr2=0x7f7650072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 -- 172.21.15.12:0/1413667608 shutdown_connections 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.479+0000 7f765730d640 1 -- 172.21.15.12:0/1413667608 wait complete. 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.480+0000 7f765730d640 1 Processor -- start 2024-03-08T08:30:08.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.480+0000 7f765730d640 1 -- start start 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76500d1110 0x7f76500c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76500d31c0 0x7f76500ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500b9980 con 0x7f765015ddc0 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500b8d50 con 0x7f76500d1110 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76500b7660 con 0x7f76500d31c0 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49896/0 (socket says 172.21.15.12:49896) 2024-03-08T08:30:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 -- 172.21.15.12:0/183485511 learned_addr learned my addr 172.21.15.12:0/183485511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76550a9640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76500d1110 0x7f76500c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 -- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76500d31c0 msgr2=0x7f76500ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76500d31c0 0x7f76500ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 -- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76500d1110 msgr2=0x7f76500c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76500d1110 0x7f76500c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 -- 172.21.15.12:0/183485511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76501610a0 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f76558aa640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7648009fa0 tx=0x7f764800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764801a070 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648003bb0 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648004030 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.481+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765014cf30 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.482+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765014d4f0 con 0x7f765015ddc0 2024-03-08T08:30:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.483+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76500749c0 con 0x7f765015ddc0 2024-03-08T08:30:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.483+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f764801d070 con 0x7f765015ddc0 2024-03-08T08:30:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.483+0000 7f7645ffb640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 0x7f763406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.484+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f764808a130 con 0x7f765015ddc0 2024-03-08T08:30:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.484+0000 7f76550a9640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 0x7f763406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.484+0000 7f76550a9640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 0x7f763406d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f764c000f60 tx=0x7f764c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.488+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7648010070 con 0x7f765015ddc0 2024-03-08T08:30:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.737+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f76501615e0 con 0x7f765015ddc0 2024-03-08T08:30:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.741+0000 7f7645ffb640 1 -- 172.21.15.12:0/183485511 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7648059390 con 0x7f765015ddc0 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 msgr2=0x7f763406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 0x7f763406d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f764c000f60 tx=0x7f764c009040 comp rx=0 tx=0).stop 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 msgr2=0x7f76500ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7648009fa0 tx=0x7f764800d560 comp rx=0 tx=0).stop 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 shutdown_connections 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f763406b4d0 0x7f763406d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765015ddc0 0x7f76500ad720 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76500d31c0 0x7f76500ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 --2- 172.21.15.12:0/183485511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76500d1110 0x7f76500c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 >> 172.21.15.12:0/183485511 conn(0x7f7650072720 msgr2=0x7f76500b7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 shutdown_connections 2024-03-08T08:30:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.743+0000 7f765730d640 1 -- 172.21.15.12:0/183485511 wait complete. 2024-03-08T08:30:08.755 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:30:08.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.855+0000 7fc2b476c640 1 Processor -- start 2024-03-08T08:30:08.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.855+0000 7fc2b476c640 1 -- start start 2024-03-08T08:30:08.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b476c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b476c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 0x7fc2ac0cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b476c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ac0cc820 con 0x7fc2ac0cf9b0 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b476c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ac15bf90 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b476c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ac0ccb00 con 0x7fc2ac0cf500 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2d09640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc2ac0cf9b0 0x7fc2ac0c5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48132/0 (socket says 172.21.15.12:48132) 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49902/0 (socket says 172.21.15.12:49902) 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2d09640 1 -- 172.21.15.12:0/4105627738 learned_addr learned my addr 172.21.15.12:0/4105627738 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/4105627738 >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 0x7fc2ac0cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 -- 172.21.15.12:0/4105627738 >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 msgr2=0x7fc2ac0cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:08.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 --2- 172.21.15.12:0/4105627738 >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 0x7fc2ac0cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 -- 172.21.15.12:0/4105627738 >> v1:172.21.15.12:6789/0 conn(0x7fc2ac0cf9b0 legacy=0x7fc2ac0c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b2508640 1 -- 172.21.15.12:0/4105627738 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac0ccde0 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b1506640 1 -- 172.21.15.12:0/4105627738 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002579113 0 0) 0x7fc2ac0cc820 con 0x7fc2ac0cf9b0 2024-03-08T08:30:08.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.856+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/4105627738 >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 0x7fc2ac0cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.857+0000 7fc2b2508640 1 --2- 172.21.15.12:0/4105627738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac072690 tx=0x7fc2a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcbd0c2bcbee61b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.857+0000 7fc2b1506640 1 -- 172.21.15.12:0/4105627738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8035020 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.857+0000 7fc2b1506640 1 -- 172.21.15.12:0/4105627738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8031830 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.857+0000 7fc2b1506640 1 -- 172.21.15.12:0/4105627738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8031c00 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 -- 172.21.15.12:0/4105627738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac15bbb0 msgr2=0x7fc2ac0cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 --2- 172.21.15.12:0/4105627738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac072690 tx=0x7fc2a802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 -- 172.21.15.12:0/4105627738 shutdown_connections 2024-03-08T08:30:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 --2- 172.21.15.12:0/4105627738 >> v2:172.21.15.156:3300/0 conn(0x7fc2ac0cf500 0x7fc2ac0cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 --2- 172.21.15.12:0/4105627738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac0cd8c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 -- 172.21.15.12:0/4105627738 >> 172.21.15.12:0/4105627738 conn(0x7fc2ac0b3420 msgr2=0x7fc2ac0b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 -- 172.21.15.12:0/4105627738 shutdown_connections 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.858+0000 7fc2b476c640 1 -- 172.21.15.12:0/4105627738 wait complete. 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.859+0000 7fc2b476c640 1 Processor -- start 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.859+0000 7fc2b476c640 1 -- start start 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 0x7fc2ac0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cde00 con 0x7fc2ac0cf500 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cd3a0 con 0x7fc2ac15bbb0 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b476c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cc440 con 0x7fc2ac0cf9b0 2024-03-08T08:30:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49918/0 (socket says 172.21.15.12:49918) 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 -- 172.21.15.12:0/3114443211 learned_addr learned my addr 172.21.15.12:0/3114443211 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2d09640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 0x7fc2ac0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 -- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 msgr2=0x7fc2ac0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 0x7fc2ac0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 -- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 msgr2=0x7fc2ac159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 -- 172.21.15.12:0/3114443211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac0c94e0 con 0x7fc2ac0cf500 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2d09640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 0x7fc2ac0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.860+0000 7fc2b2508640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc2a802bee0 tx=0x7fc2a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.861+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8041070 con 0x7fc2ac0cf500 2024-03-08T08:30:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.861+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a803f070 con 0x7fc2ac0cf500 2024-03-08T08:30:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.861+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ac0c97c0 con 0x7fc2ac0cf500 2024-03-08T08:30:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.861+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8035040 con 0x7fc2ac0cf500 2024-03-08T08:30:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.861+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac0c9d80 con 0x7fc2ac0cf500 2024-03-08T08:30:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.862+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac0749c0 con 0x7fc2ac0cf500 2024-03-08T08:30:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.862+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2a8038020 con 0x7fc2ac0cf500 2024-03-08T08:30:08.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.863+0000 7fc2a2ffd640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 0x7fc29006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:08.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.863+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 0x7fc29006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:08.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.863+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc2a80ab5e0 con 0x7fc2ac0cf500 2024-03-08T08:30:08.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.863+0000 7fc2b1d07640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 0x7fc29006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc29c006fd0 tx=0x7fc29c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:08.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:08.867+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc2a807a720 con 0x7fc2ac0cf500 2024-03-08T08:30:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.050+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2ac0d1710 con 0x7fc2ac0cf500 2024-03-08T08:30:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.051+0000 7fc2a2ffd640 1 -- 172.21.15.12:0/3114443211 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12891 (secure 0 0 0) 0x7fc2a807a020 con 0x7fc2ac0cf500 2024-03-08T08:30:09.052 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:09.052 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:06.064060+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:30:09.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 msgr2=0x7fc29006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 0x7fc29006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc29c006fd0 tx=0x7fc29c008040 comp rx=0 tx=0).stop 2024-03-08T08:30:09.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 msgr2=0x7fc2ac14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc2a802bee0 tx=0x7fc2a80046c0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 shutdown_connections 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc29006b500 0x7fc29006d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2ac15bbb0 0x7fc2ac159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2ac0cf9b0 0x7fc2ac0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 --2- 172.21.15.12:0/3114443211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ac0cf500 0x7fc2ac14eea0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 >> 172.21.15.12:0/3114443211 conn(0x7fc2ac0b3420 msgr2=0x7fc2ac0cfe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 shutdown_connections 2024-03-08T08:30:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.053+0000 7fc2b476c640 1 -- 172.21.15.12:0/3114443211 wait complete. 2024-03-08T08:30:09.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:30:09.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:30:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.168+0000 7f4409c10640 1 Processor -- start 2024-03-08T08:30:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.168+0000 7f4409c10640 1 -- start start 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440415aa30 0x7f440415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f440406d700 0x7f44040ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f440406db60 0x7f44040cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44040b2ca0 con 0x7f440406d700 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44040b2f80 con 0x7f440406db60 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4409c10640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44040b3260 con 0x7f440415aa30 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4402ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f440406d700 0x7f44040ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4402ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f440406d700 0x7f44040ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49932/0 (socket says 172.21.15.12:49932) 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.169+0000 7f4402ffd640 1 -- 172.21.15.12:0/2739578088 learned_addr learned my addr 172.21.15.12:0/2739578088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4403fff640 1 --2- 172.21.15.12:0/2739578088 >> v2:172.21.15.156:3300/0 conn(0x7f440406db60 0x7f44040cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f44037fe640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440415aa30 0x7f440415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 -- 172.21.15.12:0/2739578088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440415aa30 msgr2=0x7f440415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440415aa30 0x7f440415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 -- 172.21.15.12:0/2739578088 >> v2:172.21.15.156:3300/0 conn(0x7f440406db60 msgr2=0x7f44040cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 --2- 172.21.15.12:0/2739578088 >> v2:172.21.15.156:3300/0 conn(0x7f440406db60 0x7f44040cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 -- 172.21.15.12:0/2739578088 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44040b3540 con 0x7f440406d700 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.170+0000 7f4402ffd640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440406d700 0x7f44040ccef0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f440406f520 tx=0x7f43f802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732f414a7a15476 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f44027fc640 1 -- 172.21.15.12:0/2739578088 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f803b070 con 0x7f440406d700 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f44027fc640 1 -- 172.21.15.12:0/2739578088 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8036440 con 0x7f440406d700 2024-03-08T08:30:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f44027fc640 1 -- 172.21.15.12:0/2739578088 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8036810 con 0x7f440406d700 2024-03-08T08:30:09.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 -- 172.21.15.12:0/2739578088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440406d700 msgr2=0x7f44040ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440406d700 0x7f44040ccef0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f440406f520 tx=0x7f43f802e0e0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 -- 172.21.15.12:0/2739578088 shutdown_connections 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 --2- 172.21.15.12:0/2739578088 >> v2:172.21.15.156:3300/0 conn(0x7f440406db60 0x7f44040cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440406d700 0x7f44040ccef0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 --2- 172.21.15.12:0/2739578088 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440415aa30 0x7f440415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 -- 172.21.15.12:0/2739578088 >> 172.21.15.12:0/2739578088 conn(0x7f440406abd0 msgr2=0x7f440406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 -- 172.21.15.12:0/2739578088 shutdown_connections 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.171+0000 7f4409c10640 1 -- 172.21.15.12:0/2739578088 wait complete. 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.172+0000 7f4409c10640 1 Processor -- start 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.172+0000 7f4409c10640 1 -- start start 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 0x7f44041f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440406db60 0x7f4404158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440415e530 con 0x7f440415aa30 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440406b730 con 0x7f440406d700 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4409c10640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440415e6b0 con 0x7f440406db60 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4403fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4403fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49946/0 (socket says 172.21.15.12:49946) 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4403fff640 1 -- 172.21.15.12:0/881033914 learned_addr learned my addr 172.21.15.12:0/881033914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4403fff640 1 -- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440406db60 msgr2=0x7f4404158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f44037fe640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 0x7f44041f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.173+0000 7f4403fff640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440406db60 0x7f4404158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4403fff640 1 -- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 msgr2=0x7f44041f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4403fff640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 0x7f44041f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4403fff640 1 -- 172.21.15.12:0/881033914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440415b140 con 0x7f440415aa30 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f44037fe640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 0x7f44041f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4403fff640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007e40 tx=0x7f43fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc014620 con 0x7f440415aa30 2024-03-08T08:30:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc019070 con 0x7f440415aa30 2024-03-08T08:30:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440415b420 con 0x7f440415aa30 2024-03-08T08:30:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc014a20 con 0x7f440415aa30 2024-03-08T08:30:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.174+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440415b9e0 con 0x7f440415aa30 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.175+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44040749c0 con 0x7f440415aa30 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.175+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43fc014bc0 con 0x7f440415aa30 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.176+0000 7f43ebfff640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 0x7f43f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.176+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43fc08a780 con 0x7f440415aa30 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.176+0000 7f44037fe640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 0x7f43f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.176+0000 7f44037fe640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 0x7f43f006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.180+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f43fc059950 con 0x7f440415aa30 2024-03-08T08:30:09.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-08T08:30:09.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.359+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f440415bdc0 con 0x7f440415aa30 2024-03-08T08:30:09.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.360+0000 7f43ebfff640 1 -- 172.21.15.12:0/881033914 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f440415bdc0 con 0x7f440415aa30 2024-03-08T08:30:09.361 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:30:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 msgr2=0x7f43f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 0x7f43f006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4008040 comp rx=0 tx=0).stop 2024-03-08T08:30:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 msgr2=0x7f44040b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007e40 tx=0x7f43fc00e4b0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 shutdown_connections 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f43f006b4d0 0x7f43f006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f440415aa30 0x7f44040b06e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f440406db60 0x7f4404158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 --2- 172.21.15.12:0/881033914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f440406d700 0x7f44041f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 >> 172.21.15.12:0/881033914 conn(0x7f440406abd0 msgr2=0x7f44040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.361+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 shutdown_connections 2024-03-08T08:30:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.362+0000 7f4409c10640 1 -- 172.21.15.12:0/881033914 wait complete. 2024-03-08T08:30:09.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-08T08:30:09.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 Processor -- start 2024-03-08T08:30:09.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 -- start start 2024-03-08T08:30:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f15d815f200 0x7f15d815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15d80b14d0 0x7f15d80b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d80b2df0 con 0x7f15d80b2830 2024-03-08T08:30:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d8099b90 con 0x7f15d80b14d0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15de407640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d80cff50 con 0x7f15d815f200 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.475+0000 7f15dc9a4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15d80b14d0 0x7f15d80b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15d80b14d0 0x7f15d80b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49954/0 (socket says 172.21.15.12:49954) 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15d77fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f15d80b2830 0x7f15d815fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48148/0 (socket says 172.21.15.12:48148) 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 -- 172.21.15.12:0/557512269 learned_addr learned my addr 172.21.15.12:0/557512269 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15d7fff640 1 --2- 172.21.15.12:0/557512269 >> v2:172.21.15.156:3300/0 conn(0x7f15d815f200 0x7f15d815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 -- 172.21.15.12:0/557512269 >> v2:172.21.15.156:3300/0 conn(0x7f15d815f200 msgr2=0x7f15d815f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 --2- 172.21.15.12:0/557512269 >> v2:172.21.15.156:3300/0 conn(0x7f15d815f200 0x7f15d815f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 -- 172.21.15.12:0/557512269 >> v1:172.21.15.12:6789/0 conn(0x7f15d80b2830 legacy=0x7f15d815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15dc9a4640 1 -- 172.21.15.12:0/557512269 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d80d01a0 con 0x7f15d80b14d0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.476+0000 7f15d6ffd640 1 -- 172.21.15.12:0/557512269 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339195562 0 0) 0x7f15d80b2df0 con 0x7f15d80b2830 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15dc9a4640 1 --2- 172.21.15.12:0/557512269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d80b14d0 0x7f15d80b18b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f15d806f5f0 tx=0x7f15d002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb45db8dc18ea2de server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15d6ffd640 1 -- 172.21.15.12:0/557512269 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d002cb20 con 0x7f15d80b14d0 2024-03-08T08:30:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15d6ffd640 1 -- 172.21.15.12:0/557512269 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d002ccc0 con 0x7f15d80b14d0 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15d6ffd640 1 -- 172.21.15.12:0/557512269 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0034610 con 0x7f15d80b14d0 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 -- 172.21.15.12:0/557512269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d80b14d0 msgr2=0x7f15d80b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 --2- 172.21.15.12:0/557512269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d80b14d0 0x7f15d80b18b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f15d806f5f0 tx=0x7f15d002bf80 comp rx=0 tx=0).stop 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 -- 172.21.15.12:0/557512269 shutdown_connections 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 --2- 172.21.15.12:0/557512269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d80b14d0 0x7f15d80b18b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 --2- 172.21.15.12:0/557512269 >> v2:172.21.15.156:3300/0 conn(0x7f15d815f200 0x7f15d815f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 -- 172.21.15.12:0/557512269 >> 172.21.15.12:0/557512269 conn(0x7f15d80b3460 msgr2=0x7f15d80b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 -- 172.21.15.12:0/557512269 shutdown_connections 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.477+0000 7f15de407640 1 -- 172.21.15.12:0/557512269 wait complete. 2024-03-08T08:30:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.478+0000 7f15de407640 1 Processor -- start 2024-03-08T08:30:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 -- start start 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15d80b14d0 0x7f15d814e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 0x7f15d80c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d815f5e0 con 0x7f15d815f200 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d80d07f0 con 0x7f15d80b2830 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15de407640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d80cd720 con 0x7f15d80b14d0 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15dc9a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15dc9a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49968/0 (socket says 172.21.15.12:49968) 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.479+0000 7f15dc9a4640 1 -- 172.21.15.12:0/764916028 learned_addr learned my addr 172.21.15.12:0/764916028 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d77fe640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 0x7f15d80c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 -- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15d80b14d0 msgr2=0x7f15d814e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d7fff640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15d80b14d0 0x7f15d814e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15d80b14d0 0x7f15d814e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 -- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 msgr2=0x7f15d80c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 0x7f15d80c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 -- 172.21.15.12:0/764916028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d80c9660 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d77fe640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 0x7f15d80c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15dc9a4640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f15d0036040 tx=0x7f15d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0043020 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d003e450 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d80b6ee0 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0009510 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.480+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d80b7490 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.481+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d80749c0 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.482+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15d0046020 con 0x7f15d815f200 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.482+0000 7f15d4ff9640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 0x7f15c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.482+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f15d00ac720 con 0x7f15d815f200 2024-03-08T08:30:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.482+0000 7f15d7fff640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 0x7f15c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.483+0000 7f15d7fff640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 0x7f15c406d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f15c8006fd0 tx=0x7f15c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.487+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f15d004d030 con 0x7f15d815f200 2024-03-08T08:30:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.668+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f15d80cdd80 con 0x7f15d815f200 2024-03-08T08:30:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.668+0000 7f15d4ff9640 1 -- 172.21.15.12:0/764916028 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f15d007ba10 con 0x7f15d815f200 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:09.671 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 msgr2=0x7f15c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 0x7f15c406d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f15c8006fd0 tx=0x7f15c8008040 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 msgr2=0x7f15d81f4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f15d0036040 tx=0x7f15d0004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 shutdown_connections 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f15c406b4d0 0x7f15c406d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15d815f200 0x7f15d81f4ea0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15d80b2830 0x7f15d80c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 --2- 172.21.15.12:0/764916028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15d80b14d0 0x7f15d814e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 >> 172.21.15.12:0/764916028 conn(0x7f15d80b3460 msgr2=0x7f15d80cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 shutdown_connections 2024-03-08T08:30:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.670+0000 7f15de407640 1 -- 172.21.15.12:0/764916028 wait complete. 2024-03-08T08:30:09.681 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-08T08:30:09.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-08T08:30:09.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 Processor -- start 2024-03-08T08:30:09.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 -- start start 2024-03-08T08:30:09.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 0x7f7ab80d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab80ce580 con 0x7f7ab80b29a0 2024-03-08T08:30:09.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab80ce860 con 0x7f7ab815e050 2024-03-08T08:30:09.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.786+0000 7f7abdde2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80ceb40 con 0x7f7ab81623d0 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 0x7f7ab80d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:59750/0 (socket says 172.21.15.12:59750) 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 -- 172.21.15.12:0/2786217360 learned_addr learned my addr 172.21.15.12:0/2786217360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab7fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7ab80b29a0 0x7f7ab80b5600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:57842/0 (socket says 172.21.15.12:57842) 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 0x7f7ab80d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:41870/0 (socket says 172.21.15.12:41870) 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 -- 172.21.15.12:0/2786217360 >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 msgr2=0x7f7ab80d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/2786217360 >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 0x7f7ab80d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 -- 172.21.15.12:0/2786217360 >> v1:172.21.15.156:6789/0 conn(0x7f7ab80b29a0 legacy=0x7f7ab80b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.787+0000 7f7ab6ffd640 1 -- 172.21.15.12:0/2786217360 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80cf340 con 0x7f7ab81623d0 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.788+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/2786217360 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009870 tx=0x7f7aac02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fcae1a9c21ffde0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.788+0000 7f7ab67fc640 1 -- 172.21.15.12:0/2786217360 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac035020 con 0x7f7ab81623d0 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.788+0000 7f7ab67fc640 1 -- 172.21.15.12:0/2786217360 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac031640 con 0x7f7ab81623d0 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.788+0000 7f7ab67fc640 1 -- 172.21.15.12:0/2786217360 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac02d040 con 0x7f7ab81623d0 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 -- 172.21.15.12:0/2786217360 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 msgr2=0x7f7ab80d1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 --2- 172.21.15.12:0/2786217360 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009870 tx=0x7f7aac02b8f0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 -- 172.21.15.12:0/2786217360 shutdown_connections 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 --2- 172.21.15.12:0/2786217360 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab81623d0 0x7f7ab80d1020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 --2- 172.21.15.12:0/2786217360 >> v2:172.21.15.156:3300/0 conn(0x7f7ab815e050 0x7f7ab80d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 -- 172.21.15.12:0/2786217360 >> 172.21.15.12:0/2786217360 conn(0x7f7ab806c7c0 msgr2=0x7f7ab806cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 -- 172.21.15.12:0/2786217360 shutdown_connections 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.789+0000 7f7abdde2640 1 -- 172.21.15.12:0/2786217360 wait complete. 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.790+0000 7f7abdde2640 1 Processor -- start 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.790+0000 7f7abdde2640 1 -- start start 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 0x7f7ab80c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 0x7f7ab80c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b3200 con 0x7f7ab80b29a0 2024-03-08T08:30:09.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b3080 con 0x7f7ab81623d0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7abdde2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b5d10 con 0x7f7ab815e050 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49976/0 (socket says 172.21.15.12:49976) 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 -- 172.21.15.12:0/3643284049 learned_addr learned my addr 172.21.15.12:0/3643284049 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 0x7f7ab80c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab7fff640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 0x7f7ab80c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 -- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 msgr2=0x7f7ab80c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 0x7f7ab80c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 -- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 msgr2=0x7f7ab80c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 0x7f7ab80c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab77fe640 1 -- 172.21.15.12:0/3643284049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80b91b0 con 0x7f7ab80b29a0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 0x7f7ab80c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.791+0000 7f7ab7fff640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 0x7f7ab80c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7ab77fe640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800e9b0 tx=0x7f7aa800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8017070 con 0x7f7ab80b29a0 2024-03-08T08:30:09.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa800c9a0 con 0x7f7ab80b29a0 2024-03-08T08:30:09.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab80b7c90 con 0x7f7ab80b29a0 2024-03-08T08:30:09.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa800cd90 con 0x7f7ab80b29a0 2024-03-08T08:30:09.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.792+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab80b8250 con 0x7f7ab80b29a0 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.793+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab80749c0 con 0x7f7ab80b29a0 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.793+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aa801a070 con 0x7f7ab80b29a0 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.794+0000 7f7a9ffff640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.794+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7aa8089e10 con 0x7f7ab80b29a0 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.794+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.794+0000 7f7ab6ffd640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7aac036000 tx=0x7f7aac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.798+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7aa8059070 con 0x7f7ab80b29a0 2024-03-08T08:30:09.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:09.979+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7ab80d18b0 con 0x7f7ab80b29a0 2024-03-08T08:30:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.744+0000 7f7a9ffff640 1 -- 172.21.15.12:0/3643284049 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7aa8058970 con 0x7f7ab80b29a0 2024-03-08T08:30:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.746+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 msgr2=0x7f7aa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.746+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7aac036000 tx=0x7f7aac008030 comp rx=0 tx=0).stop 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.746+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 msgr2=0x7f7ab8152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.746+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800e9b0 tx=0x7f7aa800ee70 comp rx=0 tx=0).stop 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 shutdown_connections 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab81623d0 0x7f7ab80c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab815e050 0x7f7ab80c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 --2- 172.21.15.12:0/3643284049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ab80b29a0 0x7f7ab8152dd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 >> 172.21.15.12:0/3643284049 conn(0x7f7ab806c7c0 msgr2=0x7f7ab80d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 shutdown_connections 2024-03-08T08:30:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.747+0000 7f7abdde2640 1 -- 172.21.15.12:0/3643284049 wait complete. 2024-03-08T08:30:10.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-08T08:30:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.878+0000 7fc235d1e640 1 Processor -- start 2024-03-08T08:30:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.878+0000 7fc235d1e640 1 -- start start 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc235d1e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc23015ddc0 0x7fc2300d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc235d1e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc2300d0f40 0x7fc2300d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc235d1e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23015e1a0 con 0x7fc2300d26f0 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc235d1e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2300b9030 con 0x7fc23015ddc0 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc235d1e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2300d1d20 con 0x7fc2300d0f40 2024-03-08T08:30:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc23015ddc0 0x7fc2300d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc2300d26f0 0x7fc2300d2b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48154/0 (socket says 172.21.15.12:48154) 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc23015ddc0 0x7fc2300d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49978/0 (socket says 172.21.15.12:49978) 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22ffff640 1 -- 172.21.15.12:0/3171448763 learned_addr learned my addr 172.21.15.12:0/3171448763 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22effd640 1 --2- 172.21.15.12:0/3171448763 >> v2:172.21.15.156:3300/0 conn(0x7fc2300d0f40 0x7fc2300d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 -- 172.21.15.12:0/3171448763 >> v2:172.21.15.156:3300/0 conn(0x7fc2300d0f40 msgr2=0x7fc2300d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/3171448763 >> v2:172.21.15.156:3300/0 conn(0x7fc2300d0f40 0x7fc2300d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 -- 172.21.15.12:0/3171448763 >> v1:172.21.15.12:6789/0 conn(0x7fc2300d26f0 legacy=0x7fc2300d2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22e7fc640 1 -- 172.21.15.12:0/3171448763 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700791792 0 0) 0x7fc23015e1a0 con 0x7fc2300d26f0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.879+0000 7fc22f7fe640 1 -- 172.21.15.12:0/3171448763 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300b8790 con 0x7fc23015ddc0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.880+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/3171448763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc23015ddc0 0x7fc2300d0a00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc2300b6420 tx=0x7fc22002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8273fa400e9767db server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.880+0000 7fc22e7fc640 1 -- 172.21.15.12:0/3171448763 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220035020 con 0x7fc23015ddc0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.880+0000 7fc22e7fc640 1 -- 172.21.15.12:0/3171448763 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc220031830 con 0x7fc23015ddc0 2024-03-08T08:30:10.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.880+0000 7fc22e7fc640 1 -- 172.21.15.12:0/3171448763 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220031c00 con 0x7fc23015ddc0 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 -- 172.21.15.12:0/3171448763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc23015ddc0 msgr2=0x7fc2300d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 --2- 172.21.15.12:0/3171448763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc23015ddc0 0x7fc2300d0a00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc2300b6420 tx=0x7fc22002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 -- 172.21.15.12:0/3171448763 shutdown_connections 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 --2- 172.21.15.12:0/3171448763 >> v2:172.21.15.156:3300/0 conn(0x7fc2300d0f40 0x7fc2300d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 --2- 172.21.15.12:0/3171448763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc23015ddc0 0x7fc2300d0a00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 -- 172.21.15.12:0/3171448763 >> 172.21.15.12:0/3171448763 conn(0x7fc230072720 msgr2=0x7fc230072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 -- 172.21.15.12:0/3171448763 shutdown_connections 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.881+0000 7fc235d1e640 1 -- 172.21.15.12:0/3171448763 wait complete. 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.882+0000 7fc235d1e640 1 Processor -- start 2024-03-08T08:30:10.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.882+0000 7fc235d1e640 1 -- start start 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 0x7fc2301f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23015ddc0 0x7fc2300b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300b9980 con 0x7fc2300d26f0 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300b8d50 con 0x7fc2300d0f40 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc235d1e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300d3250 con 0x7fc23015ddc0 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49990/0 (socket says 172.21.15.12:49990) 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 -- 172.21.15.12:0/123640827 learned_addr learned my addr 172.21.15.12:0/123640827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 0x7fc2301f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 -- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23015ddc0 msgr2=0x7fc2300b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22ffff640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23015ddc0 0x7fc2300b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23015ddc0 0x7fc2300b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 -- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 msgr2=0x7fc2301f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 0x7fc2301f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 -- 172.21.15.12:0/123640827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2301f4a50 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 0x7fc2301f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.883+0000 7fc22effd640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc22400d800 tx=0x7fc22400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.884+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400b6c0 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.884+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224019070 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.884+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2301f4820 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.884+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400ba80 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.884+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2301f52b0 con 0x7fc2300d26f0 2024-03-08T08:30:10.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.885+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2300749c0 con 0x7fc2300d26f0 2024-03-08T08:30:10.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.885+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc22400bc20 con 0x7fc2300d26f0 2024-03-08T08:30:10.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.885+0000 7fc217fff640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 0x7fc21c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:10.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.886+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc22408a4d0 con 0x7fc2300d26f0 2024-03-08T08:30:10.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.886+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 0x7fc21c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:10.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.886+0000 7fc22f7fe640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 0x7fc21c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc220036000 tx=0x7fc220008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:10.890+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc224059530 con 0x7fc2300d26f0 2024-03-08T08:30:11.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.071+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc2300d2240 con 0x7fc2300d26f0 2024-03-08T08:30:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.751+0000 7fc217fff640 1 -- 172.21.15.12:0/123640827 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc224058e30 con 0x7fc2300d26f0 2024-03-08T08:30:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 msgr2=0x7fc21c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 0x7fc21c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc220036000 tx=0x7fc220008030 comp rx=0 tx=0).stop 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 msgr2=0x7fc230150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc22400d800 tx=0x7fc22400dcc0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 shutdown_connections 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc21c06b4d0 0x7fc21c06d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23015ddc0 0x7fc2300b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2300d26f0 0x7fc230150df0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 --2- 172.21.15.12:0/123640827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2300d0f40 0x7fc2301f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 >> 172.21.15.12:0/123640827 conn(0x7fc230072720 msgr2=0x7fc2300c9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 shutdown_connections 2024-03-08T08:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.755+0000 7fc235d1e640 1 -- 172.21.15.12:0/123640827 wait complete. 2024-03-08T08:30:11.783 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-08T08:30:11.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.891+0000 7fa9a4fbc640 1 Processor -- start 2024-03-08T08:30:11.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.891+0000 7fa9a4fbc640 1 -- start start 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa9a4fbc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9a0072c10 0x7fa9a00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa9a4fbc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 0x7fa9a006b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa9a4fbc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a00ad820 con 0x7fa9a015df00 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa9a4fbc640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a00afd80 con 0x7fa9a0072c10 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa9a4fbc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00ada90 con 0x7fa9a006abd0 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9a0072c10 0x7fa9a00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9a0072c10 0x7fa9a00af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49998/0 (socket says 172.21.15.12:49998) 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 -- 172.21.15.12:0/1290959623 learned_addr learned my addr 172.21.15.12:0/1290959623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:11.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99ed9d640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 0x7fa9a006b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 -- 172.21.15.12:0/1290959623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 msgr2=0x7fa9a006b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:11.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 0x7fa9a006b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.892+0000 7fa99dd9b640 1 -- 172.21.15.12:0/1290959623 >> v1:172.21.15.156:6789/0 conn(0x7fa9a015df00 legacy=0x7fa9a00b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:11.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99dd9b640 1 -- 172.21.15.12:0/1290959623 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a00add70 con 0x7fa9a0072c10 2024-03-08T08:30:11.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99ed9d640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 0x7fa9a006b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99dd9b640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a00af840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf0b9802c83d6fd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99d59a640 1 -- 172.21.15.12:0/1290959623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988035020 con 0x7fa9a0072c10 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99d59a640 1 -- 172.21.15.12:0/1290959623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988031640 con 0x7fa9a0072c10 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.893+0000 7fa99d59a640 1 -- 172.21.15.12:0/1290959623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98802d040 con 0x7fa9a0072c10 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/1290959623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 msgr2=0x7fa9a00af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a00af840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b8f0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/1290959623 shutdown_connections 2024-03-08T08:30:11.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a006abd0 0x7fa9a006b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/1290959623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a00af840 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/1290959623 >> 172.21.15.12:0/1290959623 conn(0x7fa9a006c9d0 msgr2=0x7fa9a0099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:11.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/1290959623 shutdown_connections 2024-03-08T08:30:11.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.894+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/1290959623 wait complete. 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.895+0000 7fa9a4fbc640 1 Processor -- start 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.895+0000 7fa9a4fbc640 1 -- start start 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9a006abd0 0x7fa9a0160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a015df00 0x7fa9a00cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00b5c90 con 0x7fa9a0072c10 2024-03-08T08:30:11.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00ae330 con 0x7fa9a006abd0 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa9a4fbc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00ae850 con 0x7fa9a015df00 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa99dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa99dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50012/0 (socket says 172.21.15.12:50012) 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa99dd9b640 1 -- 172.21.15.12:0/3216123376 learned_addr learned my addr 172.21.15.12:0/3216123376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa99ed9d640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a015df00 0x7fa9a00cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.896+0000 7fa99e59c640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9a006abd0 0x7fa9a0160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 -- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a015df00 msgr2=0x7fa9a00cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a015df00 0x7fa9a00cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 -- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9a006abd0 msgr2=0x7fa9a0160d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9a006abd0 0x7fa9a0160d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 -- 172.21.15.12:0/3216123376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a01f4690 con 0x7fa9a0072c10 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa99dd9b640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa988000c00 tx=0x7fa9880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988040070 con 0x7fa9a0072c10 2024-03-08T08:30:11.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988039bf0 con 0x7fa9a0072c10 2024-03-08T08:30:11.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a01f4970 con 0x7fa9a0072c10 2024-03-08T08:30:11.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988035040 con 0x7fa9a0072c10 2024-03-08T08:30:11.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.897+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a01f4f30 con 0x7fa9a0072c10 2024-03-08T08:30:11.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.898+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a00749c0 con 0x7fa9a0072c10 2024-03-08T08:30:11.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.899+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98804b060 con 0x7fa9a0072c10 2024-03-08T08:30:11.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.899+0000 7fa996ffd640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:11.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.899+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa9880abad0 con 0x7fa9a0072c10 2024-03-08T08:30:11.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.899+0000 7fa99e59c640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:11.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.899+0000 7fa99e59c640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa998006fd0 tx=0x7fa998009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:11.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:11.904+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa988031610 con 0x7fa9a0072c10 2024-03-08T08:30:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.119+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa9a00cb220 con 0x7fa9a0072c10 2024-03-08T08:30:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.753+0000 7fa996ffd640 1 -- 172.21.15.12:0/3216123376 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa98807a3f0 con 0x7fa9a0072c10 2024-03-08T08:30:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 4 and data pool 5 2024-03-08T08:30:12.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.757+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 msgr2=0x7fa98006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.757+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa998006fd0 tx=0x7fa998009060 comp rx=0 tx=0).stop 2024-03-08T08:30:12.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.757+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 msgr2=0x7fa9a01f3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.757+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa988000c00 tx=0x7fa9880046c0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 shutdown_connections 2024-03-08T08:30:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9a015df00 0x7fa9a00cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9a0072c10 0x7fa9a01f3e60 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 --2- 172.21.15.12:0/3216123376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9a006abd0 0x7fa9a0160d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.758+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 >> 172.21.15.12:0/3216123376 conn(0x7fa9a006c9d0 msgr2=0x7fa9a015c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:12.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.759+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 shutdown_connections 2024-03-08T08:30:12.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.759+0000 7fa9a4fbc640 1 -- 172.21.15.12:0/3216123376 wait complete. 2024-03-08T08:30:12.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:30:12.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.894+0000 7f5d3442e640 1 Processor -- start 2024-03-08T08:30:12.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.894+0000 7f5d3442e640 1 -- start start 2024-03-08T08:30:12.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d3442e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d3442e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c15df10 0x7f5d2c161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d3442e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c161fb0 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d3442e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c161d80 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d3442e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0d1d60 con 0x7f5d2c15df10 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d321ca640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d321ca640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:41900/0 (socket says 172.21.15.12:41900) 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d321ca640 1 -- 172.21.15.12:0/1536561126 learned_addr learned my addr 172.21.15.12:0/1536561126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d319c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c15df10 0x7f5d2c161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.895+0000 7f5d329cb640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5d2c0d0e10 0x7f5d2c0b54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:57850/0 (socket says 172.21.15.12:57850) 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.896+0000 7f5d321ca640 1 -- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c15df10 msgr2=0x7f5d2c161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.896+0000 7f5d321ca640 1 --2- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c15df10 0x7f5d2c161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.896+0000 7f5d321ca640 1 -- 172.21.15.12:0/1536561126 >> v1:172.21.15.156:6789/0 conn(0x7f5d2c0d0e10 legacy=0x7f5d2c0b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.896+0000 7f5d321ca640 1 -- 172.21.15.12:0/1536561126 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c162290 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.896+0000 7f5d321ca640 1 --2- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009870 tx=0x7f5d2802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e86ede99ebf8e712 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d311c8640 1 -- 172.21.15.12:0/1536561126 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28034070 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d311c8640 1 -- 172.21.15.12:0/1536561126 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d28030840 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d311c8640 1 -- 172.21.15.12:0/1536561126 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28030c10 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 -- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c0d0a30 msgr2=0x7f5d2c0b95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009870 tx=0x7f5d2802b980 comp rx=0 tx=0).stop 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 -- 172.21.15.12:0/1536561126 shutdown_connections 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c15df10 0x7f5d2c161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1536561126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c0d0a30 0x7f5d2c0b95a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 -- 172.21.15.12:0/1536561126 >> 172.21.15.12:0/1536561126 conn(0x7f5d2c06c710 msgr2=0x7f5d2c06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 -- 172.21.15.12:0/1536561126 shutdown_connections 2024-03-08T08:30:12.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.897+0000 7f5d3442e640 1 -- 172.21.15.12:0/1536561126 wait complete. 2024-03-08T08:30:12.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.898+0000 7f5d3442e640 1 Processor -- start 2024-03-08T08:30:12.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.898+0000 7f5d3442e640 1 -- start start 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c0d0a30 0x7f5d2c1f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c15df10 0x7f5d2c0b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0d12a0 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0d1fb0 con 0x7f5d2c15df10 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d3442e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0d34e0 con 0x7f5d2c0d0a30 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d319c9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d319c9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50020/0 (socket says 172.21.15.12:50020) 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d319c9640 1 -- 172.21.15.12:0/1992838475 learned_addr learned my addr 172.21.15.12:0/1992838475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d329cb640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c15df10 0x7f5d2c0b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.899+0000 7f5d321ca640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c0d0a30 0x7f5d2c1f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 -- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c0d0a30 msgr2=0x7f5d2c1f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c0d0a30 0x7f5d2c1f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 -- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c15df10 msgr2=0x7f5d2c0b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:12.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c15df10 0x7f5d2c0b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 -- 172.21.15.12:0/1992838475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c1f8590 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d319c9640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e840 tx=0x7f5d1c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c017070 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c00c940 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c1f3f20 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00cd10 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.900+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c15e8c0 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.901+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c0749c0 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.902+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d1c01a070 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.902+0000 7f5d22ffd640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.902+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d1c089f40 con 0x7f5d2c0d0e10 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.902+0000 7f5d321ca640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.902+0000 7f5d321ca640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 0x7f5d1006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5d28036000 tx=0x7f5d28008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:12.907+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5d2c1f3f20 con 0x7f5d2c0d0e10 2024-03-08T08:30:13.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.087+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d2c0b63e0 con 0x7f5d2c0d0e10 2024-03-08T08:30:13.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.088+0000 7f5d22ffd640 1 -- 172.21.15.12:0/1992838475 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16622 (secure 0 0 0) 0x7f5d2c1f8590 con 0x7f5d2c0d0e10 2024-03-08T08:30:13.090 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:13.090 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":34,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:12.120925+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:30:13.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 msgr2=0x7f5d1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 0x7f5d1006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5d28036000 tx=0x7f5d28008030 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 msgr2=0x7f5d2c1f39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e840 tx=0x7f5d1c00ed00 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 shutdown_connections 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d2c15df10 0x7f5d2c0b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d2c0d0e10 0x7f5d2c1f39e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 --2- 172.21.15.12:0/1992838475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d2c0d0a30 0x7f5d2c1f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 >> 172.21.15.12:0/1992838475 conn(0x7f5d2c06c710 msgr2=0x7f5d2c15bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 shutdown_connections 2024-03-08T08:30:13.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.090+0000 7f5d3442e640 1 -- 172.21.15.12:0/1992838475 wait complete. 2024-03-08T08:30:13.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:30:13.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.202+0000 7f6d1411e640 1 Processor -- start 2024-03-08T08:30:13.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 -- start start 2024-03-08T08:30:13.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6d0c0aee60 0x7f6d0c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c0cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d0c0cf130 0x7f6d0c0714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0c0b3410 con 0x7f6d0c0cf130 2024-03-08T08:30:13.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0c0cd5c0 con 0x7f6d0c0aee60 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d1411e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c15b270 con 0x7f6d0c0cd080 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.203+0000 7f6d126bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d0c0cf130 0x7f6d0c0714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d0c0cf130 0x7f6d0c0714c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50028/0 (socket says 172.21.15.12:50028) 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 -- 172.21.15.12:0/3628088890 learned_addr learned my addr 172.21.15.12:0/3628088890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d11eba640 1 --2- 172.21.15.12:0/3628088890 >> v2:172.21.15.156:3300/0 conn(0x7f6d0c0aee60 0x7f6d0c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d116b9640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c0cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 -- 172.21.15.12:0/3628088890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 msgr2=0x7f6d0c0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c0cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 -- 172.21.15.12:0/3628088890 >> v2:172.21.15.156:3300/0 conn(0x7f6d0c0aee60 msgr2=0x7f6d0c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 --2- 172.21.15.12:0/3628088890 >> v2:172.21.15.156:3300/0 conn(0x7f6d0c0aee60 0x7f6d0c0ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.204+0000 7f6d126bb640 1 -- 172.21.15.12:0/3628088890 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0c15b550 con 0x7f6d0c0cf130 2024-03-08T08:30:13.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d126bb640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c0714c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c072680 tx=0x7f6cfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c6507824824dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d10eb8640 1 -- 172.21.15.12:0/3628088890 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc02ca70 con 0x7f6d0c0cf130 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d10eb8640 1 -- 172.21.15.12:0/3628088890 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc02cc10 con 0x7f6d0c0cf130 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d10eb8640 1 -- 172.21.15.12:0/3628088890 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0343f0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 -- 172.21.15.12:0/3628088890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 msgr2=0x7f6d0c0714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c0714c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c072680 tx=0x7f6cfc02bed0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 -- 172.21.15.12:0/3628088890 shutdown_connections 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c0714c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 --2- 172.21.15.12:0/3628088890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c0cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 --2- 172.21.15.12:0/3628088890 >> v2:172.21.15.156:3300/0 conn(0x7f6d0c0aee60 0x7f6d0c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 -- 172.21.15.12:0/3628088890 >> 172.21.15.12:0/3628088890 conn(0x7f6d0c06abd0 msgr2=0x7f6d0c06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 -- 172.21.15.12:0/3628088890 shutdown_connections 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.205+0000 7f6d1411e640 1 -- 172.21.15.12:0/3628088890 wait complete. 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.206+0000 7f6d1411e640 1 Processor -- start 2024-03-08T08:30:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 -- start start 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c0aee60 0x7f6d0c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c0af240 con 0x7f6d0c0cf130 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c0cf5c0 con 0x7f6d0c0aee60 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d1411e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c0b4b70 con 0x7f6d0c0cd080 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d126bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d126bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50032/0 (socket says 172.21.15.12:50032) 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d126bb640 1 -- 172.21.15.12:0/792447075 learned_addr learned my addr 172.21.15.12:0/792447075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d11eba640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c0aee60 0x7f6d0c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.207+0000 7f6d116b9640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 -- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 msgr2=0x7f6d0c14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 -- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c0aee60 msgr2=0x7f6d0c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c0aee60 0x7f6d0c0ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 -- 172.21.15.12:0/792447075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0c0ae1c0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d126bb640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc036040 tx=0x7f6cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc043020 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc03e450 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0c0aced0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc009510 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.208+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0c0ad770 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.209+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0c0ad2c0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.210+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cfc046020 con 0x7f6d0c0cf130 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.210+0000 7f6d0a7fc640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 0x7f6cf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.210+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cfc0ac8f0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.210+0000 7f6d11eba640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 0x7f6cf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.211+0000 7f6d11eba640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 0x7f6cf806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c0b4170 tx=0x7f6d04009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.215+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6cfc04d030 con 0x7f6d0c0cf130 2024-03-08T08:30:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.396+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d0c0749a0 con 0x7f6d0c0cf130 2024-03-08T08:30:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.397+0000 7f6d0a7fc640 1 -- 172.21.15.12:0/792447075 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16622 (secure 0 0 0) 0x7f6cfc07b770 con 0x7f6d0c0cf130 2024-03-08T08:30:13.398 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:13.398 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":34,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:12.120925+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:30:13.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 msgr2=0x7f6cf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 0x7f6cf806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c0b4170 tx=0x7f6d04009190 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 msgr2=0x7f6d0c1f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc036040 tx=0x7f6cfc004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 shutdown_connections 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f6cf806b500 0x7f6cf806d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0c0cf130 0x7f6d0c1f4570 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6d0c0cd080 0x7f6d0c14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 --2- 172.21.15.12:0/792447075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c0aee60 0x7f6d0c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 >> 172.21.15.12:0/792447075 conn(0x7f6d0c06abd0 msgr2=0x7f6d0c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 shutdown_connections 2024-03-08T08:30:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.399+0000 7f6d1411e640 1 -- 172.21.15.12:0/792447075 wait complete. 2024-03-08T08:30:13.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-08T08:30:13.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.514+0000 7f19e44c9640 1 Processor -- start 2024-03-08T08:30:13.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 -- start start 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f19dc0b9760 0x7f19dc0b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc0b7520 con 0x7f19dc0b7080 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0b7800 con 0x7f19dc0b9760 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.515+0000 7f19e44c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b7ae0 con 0x7f19dc15dbc0 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2265640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f19dc0b9760 0x7f19dc0b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e1a64640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f19dc0b7080 0x7f19dc0d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:57854/0 (socket says 172.21.15.12:57854) 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2265640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:59828/0 (socket says 172.21.15.12:59828) 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e1a64640 1 -- 172.21.15.12:0/3397122702 learned_addr learned my addr 172.21.15.12:0/3397122702 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:13.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3397122702 >> v2:172.21.15.156:3300/0 conn(0x7f19dc0b9760 0x7f19dc0b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:41926/0 (socket says 172.21.15.12:41926) 2024-03-08T08:30:13.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 -- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 msgr2=0x7f19dc0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:13.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 -- 172.21.15.12:0/3397122702 >> v1:172.21.15.156:6789/0 conn(0x7f19dc0b7080 legacy=0x7f19dc0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2a66640 1 -- 172.21.15.12:0/3397122702 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0b7dc0 con 0x7f19dc0b9760 2024-03-08T08:30:13.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e1263640 1 -- 172.21.15.12:0/3397122702 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777705890 0 0) 0x7f19dc0b7520 con 0x7f19dc0b7080 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.516+0000 7f19e2265640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.517+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc0b85c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f906b7ba6c0fee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.517+0000 7f19e1263640 1 -- 172.21.15.12:0/3397122702 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc039070 con 0x7f19dc0b9760 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.517+0000 7f19e1263640 1 -- 172.21.15.12:0/3397122702 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc02cce0 con 0x7f19dc0b9760 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.517+0000 7f19e1263640 1 -- 172.21.15.12:0/3397122702 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc034660 con 0x7f19dc0b9760 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 -- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 msgr2=0x7f19dc0b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc0b85c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc02bf30 comp rx=0 tx=0).stop 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 -- 172.21.15.12:0/3397122702 shutdown_connections 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc0b85c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3397122702 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc15dbc0 0x7f19dc0b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 -- 172.21.15.12:0/3397122702 >> 172.21.15.12:0/3397122702 conn(0x7f19dc072720 msgr2=0x7f19dc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 -- 172.21.15.12:0/3397122702 shutdown_connections 2024-03-08T08:30:13.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.518+0000 7f19e44c9640 1 -- 172.21.15.12:0/3397122702 wait complete. 2024-03-08T08:30:13.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.519+0000 7f19e44c9640 1 Processor -- start 2024-03-08T08:30:13.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.519+0000 7f19e44c9640 1 -- start start 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc0b7080 0x7f19dc15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15dfa0 con 0x7f19dc15dbc0 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0d2220 con 0x7f19dc0b9760 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b8b00 con 0x7f19dc0b7080 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50038/0 (socket says 172.21.15.12:50038) 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 -- 172.21.15.12:0/3251195501 learned_addr learned my addr 172.21.15.12:0/3251195501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e1a64640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2265640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc0b7080 0x7f19dc15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 -- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc0b7080 msgr2=0x7f19dc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc0b7080 0x7f19dc15e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 -- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 msgr2=0x7f19dc160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 -- 172.21.15.12:0/3251195501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc15e660 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e2a66640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc044020 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc03a040 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc039050 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.520+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0cc7a0 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.521+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc0ccd60 con 0x7f19dc15dbc0 2024-03-08T08:30:13.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.521+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0749c0 con 0x7f19dc15dbc0 2024-03-08T08:30:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.522+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19cc003bb0 con 0x7f19dc15dbc0 2024-03-08T08:30:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.522+0000 7f19daffd640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.523+0000 7f19e2265640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.523+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19cc0ac630 con 0x7f19dc15dbc0 2024-03-08T08:30:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.523+0000 7f19e2265640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009560 tx=0x7f19d4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.527+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f19cc04e060 con 0x7f19dc15dbc0 2024-03-08T08:30:13.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:13.746+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f19dc0b5330 con 0x7f19dc15dbc0 2024-03-08T08:30:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.429+0000 7f19daffd640 1 -- 172.21.15.12:0/3251195501 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f19cc07aef0 con 0x7f19dc15dbc0 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 msgr2=0x7f19c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009560 tx=0x7f19d4009190 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 msgr2=0x7f19dc160c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 shutdown_connections 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19dc15dbc0 0x7f19dc160c80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19dc0b9760 0x7f19dc160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 --2- 172.21.15.12:0/3251195501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f19dc0b7080 0x7f19dc15e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.432+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 >> 172.21.15.12:0/3251195501 conn(0x7f19dc072720 msgr2=0x7f19dc0ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.433+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 shutdown_connections 2024-03-08T08:30:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.433+0000 7f19e44c9640 1 -- 172.21.15.12:0/3251195501 wait complete. 2024-03-08T08:30:14.447 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-08T08:30:14.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-08T08:30:14.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:30:14.525 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:30:14.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 Processor -- start 2024-03-08T08:30:14.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 -- start start 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc87015bcb0 0x7fc8700ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8700c5c00 0x7fc87015f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87015c090 con 0x7fc87009a9c0 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8700c6010 con 0x7fc8700c5c00 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.557+0000 7fc876683640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8700cd7a0 con 0x7fc87015bcb0 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8700c5c00 0x7fc87015f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8700c5c00 0x7fc87015f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50052/0 (socket says 172.21.15.12:50052) 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 -- 172.21.15.12:0/811830896 learned_addr learned my addr 172.21.15.12:0/811830896 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc86ffff640 1 --2- 172.21.15.12:0/811830896 >> v2:172.21.15.156:3300/0 conn(0x7fc87015bcb0 0x7fc8700ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 -- 172.21.15.12:0/811830896 >> v2:172.21.15.156:3300/0 conn(0x7fc87015bcb0 msgr2=0x7fc8700ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 --2- 172.21.15.12:0/811830896 >> v2:172.21.15.156:3300/0 conn(0x7fc87015bcb0 0x7fc8700ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 -- 172.21.15.12:0/811830896 >> v1:172.21.15.156:6789/0 conn(0x7fc87009a9c0 legacy=0x7fc8700c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.558+0000 7fc874c20640 1 -- 172.21.15.12:0/811830896 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870099c00 con 0x7fc8700c5c00 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc874c20640 1 --2- 172.21.15.12:0/811830896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8700c5c00 0x7fc87015f780 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc870072a30 tx=0x7fc86802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29220735cd872374 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc86effd640 1 -- 172.21.15.12:0/811830896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86802ca80 con 0x7fc8700c5c00 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc86effd640 1 -- 172.21.15.12:0/811830896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86802cc20 con 0x7fc8700c5c00 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc86effd640 1 -- 172.21.15.12:0/811830896 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8680343f0 con 0x7fc8700c5c00 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 -- 172.21.15.12:0/811830896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8700c5c00 msgr2=0x7fc87015f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 --2- 172.21.15.12:0/811830896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8700c5c00 0x7fc87015f780 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc870072a30 tx=0x7fc86802bee0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 -- 172.21.15.12:0/811830896 shutdown_connections 2024-03-08T08:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 --2- 172.21.15.12:0/811830896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8700c5c00 0x7fc87015f780 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 --2- 172.21.15.12:0/811830896 >> v2:172.21.15.156:3300/0 conn(0x7fc87015bcb0 0x7fc8700ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 -- 172.21.15.12:0/811830896 >> 172.21.15.12:0/811830896 conn(0x7fc8700b3490 msgr2=0x7fc8700b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:14.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 -- 172.21.15.12:0/811830896 shutdown_connections 2024-03-08T08:30:14.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.559+0000 7fc876683640 1 -- 172.21.15.12:0/811830896 wait complete. 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.561+0000 7fc876683640 1 Processor -- start 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.561+0000 7fc876683640 1 -- start start 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.561+0000 7fc876683640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc87009a9c0 0x7fc870070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc8700c5c00 0x7fc87006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87009a4a0 con 0x7fc87015bcb0 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87009a1c0 con 0x7fc87009a9c0 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8700cd4b0 con 0x7fc8700c5c00 2024-03-08T08:30:14.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50066/0 (socket says 172.21.15.12:50066) 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 -- 172.21.15.12:0/2590575193 learned_addr learned my addr 172.21.15.12:0/2590575193 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc86ffff640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc87009a9c0 0x7fc870070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc86f7fe640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc8700c5c00 0x7fc87006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 -- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc8700c5c00 msgr2=0x7fc87006d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc8700c5c00 0x7fc87006d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 -- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc87009a9c0 msgr2=0x7fc870070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc87009a9c0 0x7fc870070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 -- 172.21.15.12:0/2590575193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87015c420 con 0x7fc87015bcb0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc874c20640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc868009e20 tx=0x7fc868004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868043020 con 0x7fc87015bcb0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86803e450 con 0x7fc87015bcb0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87015c700 con 0x7fc87015bcb0 2024-03-08T08:30:14.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868009510 con 0x7fc87015bcb0 2024-03-08T08:30:14.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.562+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87015ccc0 con 0x7fc87015bcb0 2024-03-08T08:30:14.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.563+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8700749c0 con 0x7fc87015bcb0 2024-03-08T08:30:14.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.564+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc868046020 con 0x7fc87015bcb0 2024-03-08T08:30:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.564+0000 7fc86cff9640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.564+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8680ac8b0 con 0x7fc87015bcb0 2024-03-08T08:30:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.564+0000 7fc86ffff640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:14.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.565+0000 7fc86ffff640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 0x7fc85006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc8700b3380 tx=0x7fc860008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.568+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc86804d030 con 0x7fc87015bcb0 2024-03-08T08:30:14.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:14.787+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc8700cf6e0 con 0x7fc87015bcb0 2024-03-08T08:30:15.069 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-08T08:30:15.068+0000 7f4d24ce9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-08T08:30:15.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.773+0000 7fc86cff9640 1 -- 172.21.15.12:0/2590575193 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc86807b7c0 con 0x7fc87015bcb0 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 msgr2=0x7fc85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 0x7fc85006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc8700b3380 tx=0x7fc860008040 comp rx=0 tx=0).stop 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 msgr2=0x7fc87006d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc868009e20 tx=0x7fc868004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 shutdown_connections 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc87015bcb0 0x7fc87006d890 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc8700c5c00 0x7fc87006d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 --2- 172.21.15.12:0/2590575193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc87009a9c0 0x7fc870070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.776+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 >> 172.21.15.12:0/2590575193 conn(0x7fc8700b3490 msgr2=0x7fc8700c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.777+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 shutdown_connections 2024-03-08T08:30:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.777+0000 7fc876683640 1 -- 172.21.15.12:0/2590575193 wait complete. 2024-03-08T08:30:15.791 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-08T08:30:15.791 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:30:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.897+0000 7f1963ac9640 1 Processor -- start 2024-03-08T08:30:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.897+0000 7f1963ac9640 1 -- start start 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1963ac9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195c15dff0 0x7f195c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1963ac9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c09b150 0x7f195c0cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1963ac9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0d0910 con 0x7f195c099ca0 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1963ac9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0cca40 con 0x7f195c15dff0 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1963ac9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0ccd20 con 0x7f195c09b150 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195c15dff0 0x7f195c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195c15dff0 0x7f195c0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50080/0 (socket says 172.21.15.12:50080) 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1962066640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f195c099ca0 0x7f195c06f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48158/0 (socket says 172.21.15.12:48158) 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 -- 172.21.15.12:0/4142928538 learned_addr learned my addr 172.21.15.12:0/4142928538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961064640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c09b150 0x7f195c0cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 -- 172.21.15.12:0/4142928538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c09b150 msgr2=0x7f195c0cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c09b150 0x7f195c0cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 -- 172.21.15.12:0/4142928538 >> v1:172.21.15.12:6789/0 conn(0x7f195c099ca0 legacy=0x7f195c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1961865640 1 -- 172.21.15.12:0/4142928538 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0cda60 con 0x7f195c15dff0 2024-03-08T08:30:15.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.898+0000 7f1960863640 1 -- 172.21.15.12:0/4142928538 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805971235 0 0) 0x7f195c0d0910 con 0x7f195c099ca0 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.899+0000 7f1961865640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c15dff0 0x7f195c0cf4d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1954009870 tx=0x7f195402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bddb92963da1d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.899+0000 7f1960863640 1 -- 172.21.15.12:0/4142928538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954034070 con 0x7f195c15dff0 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.899+0000 7f1960863640 1 -- 172.21.15.12:0/4142928538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954030840 con 0x7f195c15dff0 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.899+0000 7f1960863640 1 -- 172.21.15.12:0/4142928538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954030c10 con 0x7f195c15dff0 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 -- 172.21.15.12:0/4142928538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c15dff0 msgr2=0x7f195c0cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c15dff0 0x7f195c0cf4d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1954009870 tx=0x7f195402b980 comp rx=0 tx=0).stop 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 -- 172.21.15.12:0/4142928538 shutdown_connections 2024-03-08T08:30:15.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c09b150 0x7f195c0cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 --2- 172.21.15.12:0/4142928538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c15dff0 0x7f195c0cf4d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 -- 172.21.15.12:0/4142928538 >> 172.21.15.12:0/4142928538 conn(0x7f195c0d1900 msgr2=0x7f195c0d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 -- 172.21.15.12:0/4142928538 shutdown_connections 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.900+0000 7f1963ac9640 1 -- 172.21.15.12:0/4142928538 wait complete. 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.901+0000 7f1963ac9640 1 Processor -- start 2024-03-08T08:30:15.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.901+0000 7f1963ac9640 1 -- start start 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c099ca0 0x7f195c150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 0x7f195c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c06fef0 con 0x7f195c0b4520 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0d05a0 con 0x7f195c0b6390 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1963ac9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c070070 con 0x7f195c099ca0 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50084/0 (socket says 172.21.15.12:50084) 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 -- 172.21.15.12:0/3167172052 learned_addr learned my addr 172.21.15.12:0/3167172052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961865640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c099ca0 0x7f195c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1962066640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 0x7f195c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 -- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c099ca0 msgr2=0x7f195c150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c099ca0 0x7f195c150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 -- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 msgr2=0x7f195c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 0x7f195c0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 -- 172.21.15.12:0/3167172052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0c9240 con 0x7f195c0b4520 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1962066640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 0x7f195c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.902+0000 7f1961064640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9e0 tx=0x7f195000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.903+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950017070 con 0x7f195c0b4520 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.903+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195000c9d0 con 0x7f195c0b4520 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.903+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c0b6d50 con 0x7f195c0b4520 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.903+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195000cdc0 con 0x7f195c0b4520 2024-03-08T08:30:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.903+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c0b72e0 con 0x7f195c0b4520 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.904+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c0749c0 con 0x7f195c0b4520 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.904+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f195001a070 con 0x7f195c0b4520 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.904+0000 7f195a7fc640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 0x7f194806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.904+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f195008a100 con 0x7f195c0b4520 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.904+0000 7f1961865640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 0x7f194806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.905+0000 7f1961865640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 0x7f194806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1954036000 tx=0x7f195403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:15.909+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f19500589c0 con 0x7f195c0b4520 2024-03-08T08:30:16.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.126+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f195c0c9520 con 0x7f195c0b4520 2024-03-08T08:30:16.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.126+0000 7f195a7fc640 1 -- 172.21.15.12:0/3167172052 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f195005ca60 con 0x7f195c0b4520 2024-03-08T08:30:16.128 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:16.128 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:30:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:30:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 msgr2=0x7f194806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 0x7f194806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1954036000 tx=0x7f195403b040 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 msgr2=0x7f195c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9e0 tx=0x7f195000eea0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 shutdown_connections 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f194806b4d0 0x7f194806d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f195c0b6390 0x7f195c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195c0b4520 0x7f195c0b49a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 --2- 172.21.15.12:0/3167172052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f195c099ca0 0x7f195c150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 >> 172.21.15.12:0/3167172052 conn(0x7f195c0d1900 msgr2=0x7f195c0703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 shutdown_connections 2024-03-08T08:30:16.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.128+0000 7f1963ac9640 1 -- 172.21.15.12:0/3167172052 wait complete. 2024-03-08T08:30:16.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:30:16.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.240+0000 7f1ff5e4e640 1 Processor -- start 2024-03-08T08:30:16.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.240+0000 7f1ff5e4e640 1 -- start start 2024-03-08T08:30:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1ff5e4e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ff009a580 0x7f1ff00cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1ff5e4e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff009a960 con 0x7f1ff006ce00 2024-03-08T08:30:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1ff5e4e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00d0730 con 0x7f1ff00d00b0 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1ff5e4e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00d0a10 con 0x7f1ff009a580 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ff009a580 0x7f1ff00cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ff009a580 0x7f1ff00cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50100/0 (socket says 172.21.15.12:50100) 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 -- 172.21.15.12:0/2988734613 learned_addr learned my addr 172.21.15.12:0/2988734613 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 -- 172.21.15.12:0/2988734613 >> v1:172.21.15.156:6789/0 conn(0x7f1ff00d00b0 legacy=0x7f1ff006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 -- 172.21.15.12:0/2988734613 >> v1:172.21.15.12:6789/0 conn(0x7f1ff006ce00 legacy=0x7f1ff00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fee7fc640 1 -- 172.21.15.12:0/2988734613 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646618516 0 0) 0x7f1ff009a960 con 0x7f1ff006ce00 2024-03-08T08:30:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.241+0000 7f1fef7fe640 1 -- 172.21.15.12:0/2988734613 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00d0cf0 con 0x7f1ff009a580 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.242+0000 7f1fef7fe640 1 --2- 172.21.15.12:0/2988734613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff009a580 0x7f1ff00cfb70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009870 tx=0x7f1fe002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805788586414c0a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.242+0000 7f1fee7fc640 1 -- 172.21.15.12:0/2988734613 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe002f840 con 0x7f1ff009a580 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.242+0000 7f1fee7fc640 1 -- 172.21.15.12:0/2988734613 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe002f9e0 con 0x7f1ff009a580 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.242+0000 7f1fee7fc640 1 -- 172.21.15.12:0/2988734613 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe002fdb0 con 0x7f1ff009a580 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/2988734613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff009a580 msgr2=0x7f1ff00cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/2988734613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff009a580 0x7f1ff00cfb70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009870 tx=0x7f1fe002ba20 comp rx=0 tx=0).stop 2024-03-08T08:30:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/2988734613 shutdown_connections 2024-03-08T08:30:16.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/2988734613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff009a580 0x7f1ff00cfb70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/2988734613 >> 172.21.15.12:0/2988734613 conn(0x7f1ff00b3420 msgr2=0x7f1ff00b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/2988734613 shutdown_connections 2024-03-08T08:30:16.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.243+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/2988734613 wait complete. 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.244+0000 7f1ff5e4e640 1 Processor -- start 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.244+0000 7f1ff5e4e640 1 -- start start 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 0x7f1ff00afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ff00d00b0 0x7f1ff0152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff006c120 con 0x7f1ff006ce00 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00d12b0 con 0x7f1ff00d00b0 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff006c2a0 con 0x7f1ff009a580 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50116/0 (socket says 172.21.15.12:50116) 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 -- 172.21.15.12:0/1433224947 learned_addr learned my addr 172.21.15.12:0/1433224947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1feffff640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ff00d00b0 0x7f1ff0152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1feeffd640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 0x7f1ff00afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 -- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 msgr2=0x7f1ff00afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 0x7f1ff00afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 -- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ff00d00b0 msgr2=0x7f1ff0152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ff00d00b0 0x7f1ff0152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 -- 172.21.15.12:0/1433224947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00b20e0 con 0x7f1ff006ce00 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1feeffd640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 0x7f1ff00afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fef7fe640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009870 tx=0x7f1fe0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0041400 con 0x7f1ff006ce00 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe0046070 con 0x7f1ff006ce00 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff00b23c0 con 0x7f1ff006ce00 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0041800 con 0x7f1ff006ce00 2024-03-08T08:30:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.245+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff00af520 con 0x7f1ff006ce00 2024-03-08T08:30:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.247+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff00749c0 con 0x7f1ff006ce00 2024-03-08T08:30:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.247+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fe0040070 con 0x7f1ff006ce00 2024-03-08T08:30:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.247+0000 7f1fd7fff640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.247+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fe00acaf0 con 0x7f1ff006ce00 2024-03-08T08:30:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.247+0000 7f1feeffd640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.248+0000 7f1feeffd640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ff015bdc0 tx=0x7f1fe4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.252+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1fe0033030 con 0x7f1ff006ce00 2024-03-08T08:30:16.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.431+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff00b42a0 con 0x7f1ff006ce00 2024-03-08T08:30:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.432+0000 7f1fd7fff640 1 -- 172.21.15.12:0/1433224947 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f1fe007b970 con 0x7f1ff006ce00 2024-03-08T08:30:16.435 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:16.435 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":35,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:13.764879+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 msgr2=0x7f1fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ff015bdc0 tx=0x7f1fe4009040 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 msgr2=0x7f1ff015e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009870 tx=0x7f1fe0004730 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 shutdown_connections 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ff00d00b0 0x7f1ff0152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ff009a580 0x7f1ff00afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 --2- 172.21.15.12:0/1433224947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ff006ce00 0x7f1ff015e910 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 >> 172.21.15.12:0/1433224947 conn(0x7f1ff00b3420 msgr2=0x7f1ff00b64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 shutdown_connections 2024-03-08T08:30:16.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.434+0000 7f1ff5e4e640 1 -- 172.21.15.12:0/1433224947 wait complete. 2024-03-08T08:30:16.447 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-08T08:30:16.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:30:16.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.546+0000 7f4c2dc15640 1 Processor -- start 2024-03-08T08:30:16.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 -- start start 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4c280d0a30 0x7f4c280b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c28161840 0x7f4c280d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c28161fb0 con 0x7f4c2815df10 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c280d1d60 con 0x7f4c28161840 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c2dc15640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c28162290 con 0x7f4c280d0a30 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c27fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c28161840 0x7f4c280d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c27fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c28161840 0x7f4c280d24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50128/0 (socket says 172.21.15.12:50128) 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.547+0000 7f4c27fff640 1 -- 172.21.15.12:0/3322257752 learned_addr learned my addr 172.21.15.12:0/3322257752 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c277fe640 1 --2- 172.21.15.12:0/3322257752 >> v2:172.21.15.156:3300/0 conn(0x7f4c280d0a30 0x7f4c280b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c27fff640 1 -- 172.21.15.12:0/3322257752 >> v2:172.21.15.156:3300/0 conn(0x7f4c280d0a30 msgr2=0x7f4c280b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c27fff640 1 --2- 172.21.15.12:0/3322257752 >> v2:172.21.15.156:3300/0 conn(0x7f4c280d0a30 0x7f4c280b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c27fff640 1 -- 172.21.15.12:0/3322257752 >> v1:172.21.15.156:6789/0 conn(0x7f4c2815df10 legacy=0x7f4c280d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c27fff640 1 -- 172.21.15.12:0/3322257752 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c28162570 con 0x7f4c28161840 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.548+0000 7f4c27fff640 1 --2- 172.21.15.12:0/3322257752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280d24d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4c2808cb00 tx=0x7f4c2002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f480a0d67fa3f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c267fc640 1 -- 172.21.15.12:0/3322257752 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2002cb20 con 0x7f4c28161840 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c267fc640 1 -- 172.21.15.12:0/3322257752 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2002ccc0 con 0x7f4c28161840 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c267fc640 1 -- 172.21.15.12:0/3322257752 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20034610 con 0x7f4c28161840 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 -- 172.21.15.12:0/3322257752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 msgr2=0x7f4c280d24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/3322257752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280d24d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4c2808cb00 tx=0x7f4c2002bf80 comp rx=0 tx=0).stop 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 -- 172.21.15.12:0/3322257752 shutdown_connections 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/3322257752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280d24d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/3322257752 >> v2:172.21.15.156:3300/0 conn(0x7f4c280d0a30 0x7f4c280b95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 -- 172.21.15.12:0/3322257752 >> 172.21.15.12:0/3322257752 conn(0x7f4c2806c710 msgr2=0x7f4c2806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 -- 172.21.15.12:0/3322257752 shutdown_connections 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.549+0000 7f4c2dc15640 1 -- 172.21.15.12:0/3322257752 wait complete. 2024-03-08T08:30:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.550+0000 7f4c2dc15640 1 Processor -- start 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 -- start start 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c280d0a30 0x7f4c28155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 0x7f4c280ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280d2a10 con 0x7f4c28161840 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28161cb0 con 0x7f4c280d0a30 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c2dc15640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280d2b90 con 0x7f4c2815df10 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50140/0 (socket says 172.21.15.12:50140) 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c27fff640 1 -- 172.21.15.12:0/1724962485 learned_addr learned my addr 172.21.15.12:0/1724962485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c277fe640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c280d0a30 0x7f4c28155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.551+0000 7f4c26ffd640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 0x7f4c280ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 -- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 msgr2=0x7f4c280ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 0x7f4c280ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 -- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c280d0a30 msgr2=0x7f4c28155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c280d0a30 0x7f4c28155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 -- 172.21.15.12:0/1724962485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c280cb840 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c26ffd640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 0x7f4c280ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c27fff640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009f40 tx=0x7f4c20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20043020 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2003e450 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c280cbb20 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20009510 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.552+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c280cc0e0 con 0x7f4c28161840 2024-03-08T08:30:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.553+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c280749c0 con 0x7f4c28161840 2024-03-08T08:30:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.553+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c20046020 con 0x7f4c28161840 2024-03-08T08:30:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.554+0000 7f4c0ffff640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 0x7f4c1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.554+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c200ac870 con 0x7f4c28161840 2024-03-08T08:30:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.554+0000 7f4c277fe640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 0x7f4c1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.554+0000 7f4c277fe640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 0x7f4c1406d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009b40 tx=0x7f4c18009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.558+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4c2004d030 con 0x7f4c28161840 2024-03-08T08:30:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.738+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c280cc450 con 0x7f4c1406b500 2024-03-08T08:30:16.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.744+0000 7f4c0ffff640 1 -- 172.21.15.12:0/1724962485 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250529 (secure 0 0 0) 0x7f4c280cc450 con 0x7f4c1406b500 2024-03-08T08:30:16.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:16.750 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:30:16.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 msgr2=0x7f4c1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 0x7f4c1406d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009b40 tx=0x7f4c18009190 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 msgr2=0x7f4c280cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009f40 tx=0x7f4c20004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 shutdown_connections 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f4c1406b500 0x7f4c1406d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c28161840 0x7f4c280cd2c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4c2815df10 0x7f4c280ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 --2- 172.21.15.12:0/1724962485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c280d0a30 0x7f4c28155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 >> 172.21.15.12:0/1724962485 conn(0x7f4c2806c710 msgr2=0x7f4c2815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 shutdown_connections 2024-03-08T08:30:16.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.747+0000 7f4c2dc15640 1 -- 172.21.15.12:0/1724962485 wait complete. 2024-03-08T08:30:16.760 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-08T08:30:15.006585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":178,"acting":178,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241124,"kb_used_data":4160,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733596,"statfs":{"total":773094113280,"available":772847202304,"internally_reserved":0,"allocated":4259840,"data_stored":2351307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.258621"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781694+0000","last_change":"2024-03-08T08:30:12.771963+0000","last_active":"2024-03-08T08:30:12.781694+0000","last_peered":"2024-03-08T08:30:12.781694+0000","last_clean":"2024-03-08T08:30:12.781694+0000","last_became_active":"2024-03-08T08:30:12.771583+0000","last_became_peered":"2024-03-08T08:30:12.771583+0000","last_unstale":"2024-03-08T08:30:12.781694+0000","last_undegraded":"2024-03-08T08:30:12.781694+0000","last_fullsized":"2024-03-08T08:30:12.781694+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:10.770026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771813+0000","last_change":"2024-03-08T08:30:12.768931+0000","last_active":"2024-03-08T08:30:13.771813+0000","last_peered":"2024-03-08T08:30:13.771813+0000","last_clean":"2024-03-08T08:30:13.771813+0000","last_became_active":"2024-03-08T08:30:12.768657+0000","last_became_peered":"2024-03-08T08:30:12.768657+0000","last_unstale":"2024-03-08T08:30:13.771813+0000","last_undegraded":"2024-03-08T08:30:13.771813+0000","last_fullsized":"2024-03-08T08:30:13.771813+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:13:03.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784141+0000","last_change":"2024-03-08T08:30:12.779832+0000","last_active":"2024-03-08T08:30:12.784141+0000","last_peered":"2024-03-08T08:30:12.784141+0000","last_clean":"2024-03-08T08:30:12.784141+0000","last_became_active":"2024-03-08T08:30:12.779569+0000","last_became_peered":"2024-03-08T08:30:12.779569+0000","last_unstale":"2024-03-08T08:30:12.784141+0000","last_undegraded":"2024-03-08T08:30:12.784141+0000","last_fullsized":"2024-03-08T08:30:12.784141+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:22:19.728810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771709+0000","last_change":"2024-03-08T08:30:12.773763+0000","last_active":"2024-03-08T08:30:13.771709+0000","last_peered":"2024-03-08T08:30:13.771709+0000","last_clean":"2024-03-08T08:30:13.771709+0000","last_became_active":"2024-03-08T08:30:12.773441+0000","last_became_peered":"2024-03-08T08:30:12.773441+0000","last_unstale":"2024-03-08T08:30:13.771709+0000","last_undegraded":"2024-03-08T08:30:13.771709+0000","last_fullsized":"2024-03-08T08:30:13.771709+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:52:59.982314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771054+0000","last_change":"2024-03-08T08:30:12.774949+0000","last_active":"2024-03-08T08:30:13.771054+0000","last_peered":"2024-03-08T08:30:13.771054+0000","last_clean":"2024-03-08T08:30:13.771054+0000","last_became_active":"2024-03-08T08:30:12.774445+0000","last_became_peered":"2024-03-08T08:30:12.774445+0000","last_unstale":"2024-03-08T08:30:13.771054+0000","last_undegraded":"2024-03-08T08:30:13.771054+0000","last_fullsized":"2024-03-08T08:30:13.771054+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:39:22.720405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784128+0000","last_change":"2024-03-08T08:30:12.778655+0000","last_active":"2024-03-08T08:30:12.784128+0000","last_peered":"2024-03-08T08:30:12.784128+0000","last_clean":"2024-03-08T08:30:12.784128+0000","last_became_active":"2024-03-08T08:30:12.778365+0000","last_became_peered":"2024-03-08T08:30:12.778365+0000","last_unstale":"2024-03-08T08:30:12.784128+0000","last_undegraded":"2024-03-08T08:30:12.784128+0000","last_fullsized":"2024-03-08T08:30:12.784128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784061+0000","last_change":"2024-03-08T08:30:12.773939+0000","last_active":"2024-03-08T08:30:12.784061+0000","last_peered":"2024-03-08T08:30:12.784061+0000","last_clean":"2024-03-08T08:30:12.784061+0000","last_became_active":"2024-03-08T08:30:12.773643+0000","last_became_peered":"2024-03-08T08:30:12.773643+0000","last_unstale":"2024-03-08T08:30:12.784061+0000","last_undegraded":"2024-03-08T08:30:12.784061+0000","last_fullsized":"2024-03-08T08:30:12.784061+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:11.855836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.783164+0000","last_change":"2024-03-08T08:30:12.778247+0000","last_active":"2024-03-08T08:30:12.783164+0000","last_peered":"2024-03-08T08:30:12.783164+0000","last_clean":"2024-03-08T08:30:12.783164+0000","last_became_active":"2024-03-08T08:30:12.777949+0000","last_became_peered":"2024-03-08T08:30:12.777949+0000","last_unstale":"2024-03-08T08:30:12.783164+0000","last_undegraded":"2024-03-08T08:30:12.783164+0000","last_fullsized":"2024-03-08T08:30:12.783164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:18.773299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771128+0000","last_change":"2024-03-08T08:30:12.771143+0000","last_active":"2024-03-08T08:30:13.771128+0000","last_peered":"2024-03-08T08:30:13.771128+0000","last_clean":"2024-03-08T08:30:13.771128+0000","last_became_active":"2024-03-08T08:30:12.770848+0000","last_became_peered":"2024-03-08T08:30:12.770848+0000","last_unstale":"2024-03-08T08:30:13.771128+0000","last_undegraded":"2024-03-08T08:30:13.771128+0000","last_fullsized":"2024-03-08T08:30:13.771128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:23:01.198137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771142+0000","last_change":"2024-03-08T08:30:12.769530+0000","last_active":"2024-03-08T08:30:12.771142+0000","last_peered":"2024-03-08T08:30:12.771142+0000","last_clean":"2024-03-08T08:30:12.771142+0000","last_became_active":"2024-03-08T08:30:12.769004+0000","last_became_peered":"2024-03-08T08:30:12.769004+0000","last_unstale":"2024-03-08T08:30:12.771142+0000","last_undegraded":"2024-03-08T08:30:12.771142+0000","last_fullsized":"2024-03-08T08:30:12.771142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781794+0000","last_change":"2024-03-08T08:30:12.771997+0000","last_active":"2024-03-08T08:30:12.781794+0000","last_peered":"2024-03-08T08:30:12.781794+0000","last_clean":"2024-03-08T08:30:12.781794+0000","last_became_active":"2024-03-08T08:30:12.771384+0000","last_became_peered":"2024-03-08T08:30:12.771384+0000","last_unstale":"2024-03-08T08:30:12.781794+0000","last_undegraded":"2024-03-08T08:30:12.781794+0000","last_fullsized":"2024-03-08T08:30:12.781794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.777421+0000","last_change":"2024-03-08T08:30:12.772353+0000","last_active":"2024-03-08T08:30:12.777421+0000","last_peered":"2024-03-08T08:30:12.777421+0000","last_clean":"2024-03-08T08:30:12.777421+0000","last_became_active":"2024-03-08T08:30:12.772053+0000","last_became_peered":"2024-03-08T08:30:12.772053+0000","last_unstale":"2024-03-08T08:30:12.777421+0000","last_undegraded":"2024-03-08T08:30:12.777421+0000","last_fullsized":"2024-03-08T08:30:12.777421+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:17:12.613908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764859+0000","last_change":"2024-03-08T08:30:12.762997+0000","last_active":"2024-03-08T08:30:12.764859+0000","last_peered":"2024-03-08T08:30:12.764859+0000","last_clean":"2024-03-08T08:30:12.764859+0000","last_became_active":"2024-03-08T08:30:12.762670+0000","last_became_peered":"2024-03-08T08:30:12.762670+0000","last_unstale":"2024-03-08T08:30:12.764859+0000","last_undegraded":"2024-03-08T08:30:12.764859+0000","last_fullsized":"2024-03-08T08:30:12.764859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771114+0000","last_change":"2024-03-08T08:30:12.786769+0000","last_active":"2024-03-08T08:30:13.771114+0000","last_peered":"2024-03-08T08:30:13.771114+0000","last_clean":"2024-03-08T08:30:13.771114+0000","last_became_active":"2024-03-08T08:30:12.786508+0000","last_became_peered":"2024-03-08T08:30:12.786508+0000","last_unstale":"2024-03-08T08:30:13.771114+0000","last_undegraded":"2024-03-08T08:30:13.771114+0000","last_fullsized":"2024-03-08T08:30:13.771114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:46:59.687919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771338+0000","last_change":"2024-03-08T08:30:12.770360+0000","last_active":"2024-03-08T08:30:13.771338+0000","last_peered":"2024-03-08T08:30:13.771338+0000","last_clean":"2024-03-08T08:30:13.771338+0000","last_became_active":"2024-03-08T08:30:12.770023+0000","last_became_peered":"2024-03-08T08:30:12.770023+0000","last_unstale":"2024-03-08T08:30:13.771338+0000","last_undegraded":"2024-03-08T08:30:13.771338+0000","last_fullsized":"2024-03-08T08:30:13.771338+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:10.770026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771312+0000","last_change":"2024-03-08T08:30:12.772061+0000","last_active":"2024-03-08T08:30:13.771312+0000","last_peered":"2024-03-08T08:30:13.771312+0000","last_clean":"2024-03-08T08:30:13.771312+0000","last_became_active":"2024-03-08T08:30:12.771756+0000","last_became_peered":"2024-03-08T08:30:12.771756+0000","last_unstale":"2024-03-08T08:30:13.771312+0000","last_undegraded":"2024-03-08T08:30:13.771312+0000","last_fullsized":"2024-03-08T08:30:13.771312+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:05:07.480553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.772721+0000","last_change":"2024-03-08T08:30:12.769148+0000","last_active":"2024-03-08T08:30:12.772721+0000","last_peered":"2024-03-08T08:30:12.772721+0000","last_clean":"2024-03-08T08:30:12.772721+0000","last_became_active":"2024-03-08T08:30:12.768864+0000","last_became_peered":"2024-03-08T08:30:12.768864+0000","last_unstale":"2024-03-08T08:30:12.772721+0000","last_undegraded":"2024-03-08T08:30:12.772721+0000","last_fullsized":"2024-03-08T08:30:12.772721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:26.996959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.778836+0000","last_change":"2024-03-08T08:30:12.773425+0000","last_active":"2024-03-08T08:30:12.778836+0000","last_peered":"2024-03-08T08:30:12.778836+0000","last_clean":"2024-03-08T08:30:12.778836+0000","last_became_active":"2024-03-08T08:30:12.772885+0000","last_became_peered":"2024-03-08T08:30:12.772885+0000","last_unstale":"2024-03-08T08:30:12.778836+0000","last_undegraded":"2024-03-08T08:30:12.778836+0000","last_fullsized":"2024-03-08T08:30:12.778836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:18.773299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788520+0000","last_change":"2024-03-08T08:30:12.785148+0000","last_active":"2024-03-08T08:30:12.788520+0000","last_peered":"2024-03-08T08:30:12.788520+0000","last_clean":"2024-03-08T08:30:12.788520+0000","last_became_active":"2024-03-08T08:30:12.784849+0000","last_became_peered":"2024-03-08T08:30:12.784849+0000","last_unstale":"2024-03-08T08:30:12.788520+0000","last_undegraded":"2024-03-08T08:30:12.788520+0000","last_fullsized":"2024-03-08T08:30:12.788520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781030+0000","last_change":"2024-03-08T08:30:12.779346+0000","last_active":"2024-03-08T08:30:12.781030+0000","last_peered":"2024-03-08T08:30:12.781030+0000","last_clean":"2024-03-08T08:30:12.781030+0000","last_became_active":"2024-03-08T08:30:12.778776+0000","last_became_peered":"2024-03-08T08:30:12.778776+0000","last_unstale":"2024-03-08T08:30:12.781030+0000","last_undegraded":"2024-03-08T08:30:12.781030+0000","last_fullsized":"2024-03-08T08:30:12.781030+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784094+0000","last_change":"2024-03-08T08:30:12.774559+0000","last_active":"2024-03-08T08:30:12.784094+0000","last_peered":"2024-03-08T08:30:12.784094+0000","last_clean":"2024-03-08T08:30:12.784094+0000","last_became_active":"2024-03-08T08:30:12.774290+0000","last_became_peered":"2024-03-08T08:30:12.774290+0000","last_unstale":"2024-03-08T08:30:12.784094+0000","last_undegraded":"2024-03-08T08:30:12.784094+0000","last_fullsized":"2024-03-08T08:30:12.784094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:15.001228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782740+0000","last_change":"2024-03-08T08:30:12.776173+0000","last_active":"2024-03-08T08:30:12.782740+0000","last_peered":"2024-03-08T08:30:12.782740+0000","last_clean":"2024-03-08T08:30:12.782740+0000","last_became_active":"2024-03-08T08:30:12.775895+0000","last_became_peered":"2024-03-08T08:30:12.775895+0000","last_unstale":"2024-03-08T08:30:12.782740+0000","last_undegraded":"2024-03-08T08:30:12.782740+0000","last_fullsized":"2024-03-08T08:30:12.782740+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:17:12.613908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784503+0000","last_change":"2024-03-08T08:30:12.783252+0000","last_active":"2024-03-08T08:30:12.784503+0000","last_peered":"2024-03-08T08:30:12.784503+0000","last_clean":"2024-03-08T08:30:12.784503+0000","last_became_active":"2024-03-08T08:30:12.782968+0000","last_became_peered":"2024-03-08T08:30:12.782968+0000","last_unstale":"2024-03-08T08:30:12.784503+0000","last_undegraded":"2024-03-08T08:30:12.784503+0000","last_fullsized":"2024-03-08T08:30:12.784503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:22:36.472741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771644+0000","last_change":"2024-03-08T08:30:12.767160+0000","last_active":"2024-03-08T08:30:13.771644+0000","last_peered":"2024-03-08T08:30:13.771644+0000","last_clean":"2024-03-08T08:30:13.771644+0000","last_became_active":"2024-03-08T08:30:12.766783+0000","last_became_peered":"2024-03-08T08:30:12.766783+0000","last_unstale":"2024-03-08T08:30:13.771644+0000","last_undegraded":"2024-03-08T08:30:13.771644+0000","last_fullsized":"2024-03-08T08:30:13.771644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:03:00.953042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784245+0000","last_change":"2024-03-08T08:30:12.774300+0000","last_active":"2024-03-08T08:30:12.784245+0000","last_peered":"2024-03-08T08:30:12.784245+0000","last_clean":"2024-03-08T08:30:12.784245+0000","last_became_active":"2024-03-08T08:30:12.773829+0000","last_became_peered":"2024-03-08T08:30:12.773829+0000","last_unstale":"2024-03-08T08:30:12.784245+0000","last_undegraded":"2024-03-08T08:30:12.784245+0000","last_fullsized":"2024-03-08T08:30:12.784245+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:26.996959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.778812+0000","last_change":"2024-03-08T08:30:12.773181+0000","last_active":"2024-03-08T08:30:12.778812+0000","last_peered":"2024-03-08T08:30:12.778812+0000","last_clean":"2024-03-08T08:30:12.778812+0000","last_became_active":"2024-03-08T08:30:12.772567+0000","last_became_peered":"2024-03-08T08:30:12.772567+0000","last_unstale":"2024-03-08T08:30:12.778812+0000","last_undegraded":"2024-03-08T08:30:12.778812+0000","last_fullsized":"2024-03-08T08:30:12.778812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:13:35.419025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771213+0000","last_change":"2024-03-08T08:30:12.770045+0000","last_active":"2024-03-08T08:30:12.771213+0000","last_peered":"2024-03-08T08:30:12.771213+0000","last_clean":"2024-03-08T08:30:12.771213+0000","last_became_active":"2024-03-08T08:30:12.769707+0000","last_became_peered":"2024-03-08T08:30:12.769707+0000","last_unstale":"2024-03-08T08:30:12.771213+0000","last_undegraded":"2024-03-08T08:30:12.771213+0000","last_fullsized":"2024-03-08T08:30:12.771213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781201+0000","last_change":"2024-03-08T08:30:12.773265+0000","last_active":"2024-03-08T08:30:12.781201+0000","last_peered":"2024-03-08T08:30:12.781201+0000","last_clean":"2024-03-08T08:30:12.781201+0000","last_became_active":"2024-03-08T08:30:12.772979+0000","last_became_peered":"2024-03-08T08:30:12.772979+0000","last_unstale":"2024-03-08T08:30:12.781201+0000","last_undegraded":"2024-03-08T08:30:12.781201+0000","last_fullsized":"2024-03-08T08:30:12.781201+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:36.852386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770760+0000","last_change":"2024-03-08T08:30:12.774695+0000","last_active":"2024-03-08T08:30:13.770760+0000","last_peered":"2024-03-08T08:30:13.770760+0000","last_clean":"2024-03-08T08:30:13.770760+0000","last_became_active":"2024-03-08T08:30:12.774417+0000","last_became_peered":"2024-03-08T08:30:12.774417+0000","last_unstale":"2024-03-08T08:30:13.770760+0000","last_undegraded":"2024-03-08T08:30:13.770760+0000","last_fullsized":"2024-03-08T08:30:13.770760+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:37:31.325473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.780665+0000","last_change":"2024-03-08T08:30:12.778985+0000","last_active":"2024-03-08T08:30:12.780665+0000","last_peered":"2024-03-08T08:30:12.780665+0000","last_clean":"2024-03-08T08:30:12.780665+0000","last_became_active":"2024-03-08T08:30:12.778722+0000","last_became_peered":"2024-03-08T08:30:12.778722+0000","last_unstale":"2024-03-08T08:30:12.780665+0000","last_undegraded":"2024-03-08T08:30:12.780665+0000","last_fullsized":"2024-03-08T08:30:12.780665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771228+0000","last_change":"2024-03-08T08:30:11.772402+0000","last_active":"2024-03-08T08:30:13.771228+0000","last_peered":"2024-03-08T08:30:13.771228+0000","last_clean":"2024-03-08T08:30:13.771228+0000","last_became_active":"2024-03-08T08:30:11.772090+0000","last_became_peered":"2024-03-08T08:30:11.772090+0000","last_unstale":"2024-03-08T08:30:13.771228+0000","last_undegraded":"2024-03-08T08:30:13.771228+0000","last_fullsized":"2024-03-08T08:30:13.771228+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:22:35.470453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781668+0000","last_change":"2024-03-08T08:30:12.771686+0000","last_active":"2024-03-08T08:30:12.781668+0000","last_peered":"2024-03-08T08:30:12.781668+0000","last_clean":"2024-03-08T08:30:12.781668+0000","last_became_active":"2024-03-08T08:30:12.770935+0000","last_became_peered":"2024-03-08T08:30:12.770935+0000","last_unstale":"2024-03-08T08:30:12.781668+0000","last_undegraded":"2024-03-08T08:30:12.781668+0000","last_fullsized":"2024-03-08T08:30:12.781668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:46.691888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.786866+0000","last_change":"2024-03-08T08:30:11.768998+0000","last_active":"2024-03-08T08:30:12.786866+0000","last_peered":"2024-03-08T08:30:12.786866+0000","last_clean":"2024-03-08T08:30:12.786866+0000","last_became_active":"2024-03-08T08:30:11.768654+0000","last_became_peered":"2024-03-08T08:30:11.768654+0000","last_unstale":"2024-03-08T08:30:12.786866+0000","last_undegraded":"2024-03-08T08:30:12.786866+0000","last_fullsized":"2024-03-08T08:30:12.786866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:10.853548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784265+0000","last_change":"2024-03-08T08:30:12.774724+0000","last_active":"2024-03-08T08:30:12.784265+0000","last_peered":"2024-03-08T08:30:12.784265+0000","last_clean":"2024-03-08T08:30:12.784265+0000","last_became_active":"2024-03-08T08:30:12.774439+0000","last_became_peered":"2024-03-08T08:30:12.774439+0000","last_unstale":"2024-03-08T08:30:12.784265+0000","last_undegraded":"2024-03-08T08:30:12.784265+0000","last_fullsized":"2024-03-08T08:30:12.784265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771440+0000","last_change":"2024-03-08T08:30:11.772633+0000","last_active":"2024-03-08T08:30:13.771440+0000","last_peered":"2024-03-08T08:30:13.771440+0000","last_clean":"2024-03-08T08:30:13.771440+0000","last_became_active":"2024-03-08T08:30:11.772209+0000","last_became_peered":"2024-03-08T08:30:11.772209+0000","last_unstale":"2024-03-08T08:30:13.771440+0000","last_undegraded":"2024-03-08T08:30:13.771440+0000","last_fullsized":"2024-03-08T08:30:13.771440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:06:20.197407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781393+0000","last_change":"2024-03-08T08:30:12.773677+0000","last_active":"2024-03-08T08:30:12.781393+0000","last_peered":"2024-03-08T08:30:12.781393+0000","last_clean":"2024-03-08T08:30:12.781393+0000","last_became_active":"2024-03-08T08:30:12.773407+0000","last_became_peered":"2024-03-08T08:30:12.773407+0000","last_unstale":"2024-03-08T08:30:12.781393+0000","last_undegraded":"2024-03-08T08:30:12.781393+0000","last_fullsized":"2024-03-08T08:30:12.781393+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.785451+0000","last_change":"2024-03-08T08:30:11.768037+0000","last_active":"2024-03-08T08:30:12.785451+0000","last_peered":"2024-03-08T08:30:12.785451+0000","last_clean":"2024-03-08T08:30:12.785451+0000","last_became_active":"2024-03-08T08:30:11.767329+0000","last_became_peered":"2024-03-08T08:30:11.767329+0000","last_unstale":"2024-03-08T08:30:12.785451+0000","last_undegraded":"2024-03-08T08:30:12.785451+0000","last_fullsized":"2024-03-08T08:30:12.785451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:13.998940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788898+0000","last_change":"2024-03-08T08:30:11.772419+0000","last_active":"2024-03-08T08:30:12.788898+0000","last_peered":"2024-03-08T08:30:12.788898+0000","last_clean":"2024-03-08T08:30:12.788898+0000","last_became_active":"2024-03-08T08:30:11.772166+0000","last_became_peered":"2024-03-08T08:30:11.772166+0000","last_unstale":"2024-03-08T08:30:12.788898+0000","last_undegraded":"2024-03-08T08:30:12.788898+0000","last_fullsized":"2024-03-08T08:30:12.788898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:39:21.718117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784179+0000","last_change":"2024-03-08T08:30:12.779459+0000","last_active":"2024-03-08T08:30:12.784179+0000","last_peered":"2024-03-08T08:30:12.784179+0000","last_clean":"2024-03-08T08:30:12.784179+0000","last_became_active":"2024-03-08T08:30:12.778200+0000","last_became_peered":"2024-03-08T08:30:12.778200+0000","last_unstale":"2024-03-08T08:30:12.784179+0000","last_undegraded":"2024-03-08T08:30:12.784179+0000","last_fullsized":"2024-03-08T08:30:12.784179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788556+0000","last_change":"2024-03-08T08:30:11.770944+0000","last_active":"2024-03-08T08:30:12.788556+0000","last_peered":"2024-03-08T08:30:12.788556+0000","last_clean":"2024-03-08T08:30:12.788556+0000","last_became_active":"2024-03-08T08:30:11.770228+0000","last_became_peered":"2024-03-08T08:30:11.770228+0000","last_unstale":"2024-03-08T08:30:12.788556+0000","last_undegraded":"2024-03-08T08:30:12.788556+0000","last_fullsized":"2024-03-08T08:30:12.788556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:29:57.683266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770987+0000","last_change":"2024-03-08T08:30:11.762487+0000","last_active":"2024-03-08T08:30:13.770987+0000","last_peered":"2024-03-08T08:30:13.770987+0000","last_clean":"2024-03-08T08:30:13.770987+0000","last_became_active":"2024-03-08T08:30:11.762176+0000","last_became_peered":"2024-03-08T08:30:11.762176+0000","last_unstale":"2024-03-08T08:30:13.770987+0000","last_undegraded":"2024-03-08T08:30:13.770987+0000","last_fullsized":"2024-03-08T08:30:13.770987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:03:13.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781342+0000","last_change":"2024-03-08T08:30:12.770530+0000","last_active":"2024-03-08T08:30:12.781342+0000","last_peered":"2024-03-08T08:30:12.781342+0000","last_clean":"2024-03-08T08:30:12.781342+0000","last_became_active":"2024-03-08T08:30:12.770235+0000","last_became_peered":"2024-03-08T08:30:12.770235+0000","last_unstale":"2024-03-08T08:30:12.781342+0000","last_undegraded":"2024-03-08T08:30:12.781342+0000","last_fullsized":"2024-03-08T08:30:12.781342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.763191+0000","last_change":"2024-03-08T08:30:11.775419+0000","last_active":"2024-03-08T08:30:12.763191+0000","last_peered":"2024-03-08T08:30:12.763191+0000","last_clean":"2024-03-08T08:30:12.763191+0000","last_became_active":"2024-03-08T08:30:11.775074+0000","last_became_peered":"2024-03-08T08:30:11.775074+0000","last_unstale":"2024-03-08T08:30:12.763191+0000","last_undegraded":"2024-03-08T08:30:12.763191+0000","last_fullsized":"2024-03-08T08:30:12.763191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:49:55.509413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771049+0000","last_change":"2024-03-08T08:30:12.773585+0000","last_active":"2024-03-08T08:30:13.771049+0000","last_peered":"2024-03-08T08:30:13.771049+0000","last_clean":"2024-03-08T08:30:13.771049+0000","last_became_active":"2024-03-08T08:30:12.773306+0000","last_became_peered":"2024-03-08T08:30:12.773306+0000","last_unstale":"2024-03-08T08:30:13.771049+0000","last_undegraded":"2024-03-08T08:30:13.771049+0000","last_fullsized":"2024-03-08T08:30:13.771049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:07:40.421935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.772067+0000","last_change":"2024-03-08T08:30:11.770207+0000","last_active":"2024-03-08T08:30:13.772067+0000","last_peered":"2024-03-08T08:30:13.772067+0000","last_clean":"2024-03-08T08:30:13.772067+0000","last_became_active":"2024-03-08T08:30:11.769935+0000","last_became_peered":"2024-03-08T08:30:11.769935+0000","last_unstale":"2024-03-08T08:30:13.772067+0000","last_undegraded":"2024-03-08T08:30:13.772067+0000","last_fullsized":"2024-03-08T08:30:13.772067+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:35:56.043443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781076+0000","last_change":"2024-03-08T08:30:12.769250+0000","last_active":"2024-03-08T08:30:12.781076+0000","last_peered":"2024-03-08T08:30:12.781076+0000","last_clean":"2024-03-08T08:30:12.781076+0000","last_became_active":"2024-03-08T08:30:12.768740+0000","last_became_peered":"2024-03-08T08:30:12.768740+0000","last_unstale":"2024-03-08T08:30:12.781076+0000","last_undegraded":"2024-03-08T08:30:12.781076+0000","last_fullsized":"2024-03-08T08:30:12.781076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771770+0000","last_change":"2024-03-08T08:30:11.774201+0000","last_active":"2024-03-08T08:30:13.771770+0000","last_peered":"2024-03-08T08:30:13.771770+0000","last_clean":"2024-03-08T08:30:13.771770+0000","last_became_active":"2024-03-08T08:30:11.773794+0000","last_became_peered":"2024-03-08T08:30:11.773794+0000","last_unstale":"2024-03-08T08:30:13.771770+0000","last_undegraded":"2024-03-08T08:30:13.771770+0000","last_fullsized":"2024-03-08T08:30:13.771770+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:11:09.060145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784101+0000","last_change":"2024-03-08T08:30:12.778402+0000","last_active":"2024-03-08T08:30:12.784101+0000","last_peered":"2024-03-08T08:30:12.784101+0000","last_clean":"2024-03-08T08:30:12.784101+0000","last_became_active":"2024-03-08T08:30:12.777928+0000","last_became_peered":"2024-03-08T08:30:12.777928+0000","last_unstale":"2024-03-08T08:30:12.784101+0000","last_undegraded":"2024-03-08T08:30:12.784101+0000","last_fullsized":"2024-03-08T08:30:12.784101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:49:34.108392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771100+0000","last_change":"2024-03-08T08:30:11.767315+0000","last_active":"2024-03-08T08:30:13.771100+0000","last_peered":"2024-03-08T08:30:13.771100+0000","last_clean":"2024-03-08T08:30:13.771100+0000","last_became_active":"2024-03-08T08:30:11.766482+0000","last_became_peered":"2024-03-08T08:30:11.766482+0000","last_unstale":"2024-03-08T08:30:13.771100+0000","last_undegraded":"2024-03-08T08:30:13.771100+0000","last_fullsized":"2024-03-08T08:30:13.771100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:40:43.309762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770951+0000","last_change":"2024-03-08T08:30:12.782524+0000","last_active":"2024-03-08T08:30:13.770951+0000","last_peered":"2024-03-08T08:30:13.770951+0000","last_clean":"2024-03-08T08:30:13.770951+0000","last_became_active":"2024-03-08T08:30:12.782202+0000","last_became_peered":"2024-03-08T08:30:12.782202+0000","last_unstale":"2024-03-08T08:30:13.770951+0000","last_undegraded":"2024-03-08T08:30:13.770951+0000","last_fullsized":"2024-03-08T08:30:13.770951+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:05:56.432265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.790230+0000","last_change":"2024-03-08T08:30:11.768457+0000","last_active":"2024-03-08T08:30:12.790230+0000","last_peered":"2024-03-08T08:30:12.790230+0000","last_clean":"2024-03-08T08:30:12.790230+0000","last_became_active":"2024-03-08T08:30:11.768076+0000","last_became_peered":"2024-03-08T08:30:11.768076+0000","last_unstale":"2024-03-08T08:30:12.790230+0000","last_undegraded":"2024-03-08T08:30:12.790230+0000","last_fullsized":"2024-03-08T08:30:12.790230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:35.850098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.779430+0000","last_change":"2024-03-08T08:30:12.772512+0000","last_active":"2024-03-08T08:30:12.779430+0000","last_peered":"2024-03-08T08:30:12.779430+0000","last_clean":"2024-03-08T08:30:12.779430+0000","last_became_active":"2024-03-08T08:30:12.772000+0000","last_became_peered":"2024-03-08T08:30:12.772000+0000","last_unstale":"2024-03-08T08:30:12.779430+0000","last_undegraded":"2024-03-08T08:30:12.779430+0000","last_fullsized":"2024-03-08T08:30:12.779430+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.789527+0000","last_change":"2024-03-08T08:30:11.774795+0000","last_active":"2024-03-08T08:30:12.789527+0000","last_peered":"2024-03-08T08:30:12.789527+0000","last_clean":"2024-03-08T08:30:12.789527+0000","last_became_active":"2024-03-08T08:30:11.773890+0000","last_became_peered":"2024-03-08T08:30:11.773890+0000","last_unstale":"2024-03-08T08:30:12.789527+0000","last_undegraded":"2024-03-08T08:30:12.789527+0000","last_fullsized":"2024-03-08T08:30:12.789527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:54.157869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782955+0000","last_change":"2024-03-08T08:30:12.777692+0000","last_active":"2024-03-08T08:30:12.782955+0000","last_peered":"2024-03-08T08:30:12.782955+0000","last_clean":"2024-03-08T08:30:12.782955+0000","last_became_active":"2024-03-08T08:30:12.775558+0000","last_became_peered":"2024-03-08T08:30:12.775558+0000","last_unstale":"2024-03-08T08:30:12.782955+0000","last_undegraded":"2024-03-08T08:30:12.782955+0000","last_fullsized":"2024-03-08T08:30:12.782955+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:13:35.419025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.758388+0000","last_change":"2024-03-08T08:30:11.766601+0000","last_active":"2024-03-08T08:30:12.758388+0000","last_peered":"2024-03-08T08:30:12.758388+0000","last_clean":"2024-03-08T08:30:12.758388+0000","last_became_active":"2024-03-08T08:30:11.766308+0000","last_became_peered":"2024-03-08T08:30:11.766308+0000","last_unstale":"2024-03-08T08:30:12.758388+0000","last_undegraded":"2024-03-08T08:30:12.758388+0000","last_fullsized":"2024-03-08T08:30:12.758388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:35.850098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784305+0000","last_change":"2024-03-08T08:30:12.779637+0000","last_active":"2024-03-08T08:30:12.784305+0000","last_peered":"2024-03-08T08:30:12.784305+0000","last_clean":"2024-03-08T08:30:12.784305+0000","last_became_active":"2024-03-08T08:30:12.779156+0000","last_became_peered":"2024-03-08T08:30:12.779156+0000","last_unstale":"2024-03-08T08:30:12.784305+0000","last_undegraded":"2024-03-08T08:30:12.784305+0000","last_fullsized":"2024-03-08T08:30:12.784305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.757723+0000","last_change":"2024-03-08T08:30:11.768109+0000","last_active":"2024-03-08T08:30:12.757723+0000","last_peered":"2024-03-08T08:30:12.757723+0000","last_clean":"2024-03-08T08:30:12.757723+0000","last_became_active":"2024-03-08T08:30:11.767831+0000","last_became_peered":"2024-03-08T08:30:11.767831+0000","last_unstale":"2024-03-08T08:30:12.757723+0000","last_undegraded":"2024-03-08T08:30:12.757723+0000","last_fullsized":"2024-03-08T08:30:12.757723+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:09.767738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770861+0000","last_change":"2024-03-08T08:30:12.786123+0000","last_active":"2024-03-08T08:30:13.770861+0000","last_peered":"2024-03-08T08:30:13.770861+0000","last_clean":"2024-03-08T08:30:13.770861+0000","last_became_active":"2024-03-08T08:30:12.785834+0000","last_became_peered":"2024-03-08T08:30:12.785834+0000","last_unstale":"2024-03-08T08:30:13.770861+0000","last_undegraded":"2024-03-08T08:30:13.770861+0000","last_fullsized":"2024-03-08T08:30:13.770861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:41:40.311673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770896+0000","last_change":"2024-03-08T08:30:11.766823+0000","last_active":"2024-03-08T08:30:13.770896+0000","last_peered":"2024-03-08T08:30:13.770896+0000","last_clean":"2024-03-08T08:30:13.770896+0000","last_became_active":"2024-03-08T08:30:11.766490+0000","last_became_peered":"2024-03-08T08:30:11.766490+0000","last_unstale":"2024-03-08T08:30:13.770896+0000","last_undegraded":"2024-03-08T08:30:13.770896+0000","last_fullsized":"2024-03-08T08:30:13.770896+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:48:35.326526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771101+0000","last_change":"2024-03-08T08:30:12.770278+0000","last_active":"2024-03-08T08:30:13.771101+0000","last_peered":"2024-03-08T08:30:13.771101+0000","last_clean":"2024-03-08T08:30:13.771101+0000","last_became_active":"2024-03-08T08:30:12.769978+0000","last_became_peered":"2024-03-08T08:30:12.769978+0000","last_unstale":"2024-03-08T08:30:13.771101+0000","last_undegraded":"2024-03-08T08:30:13.771101+0000","last_fullsized":"2024-03-08T08:30:13.771101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:47:29.489464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771384+0000","last_change":"2024-03-08T08:30:11.769883+0000","last_active":"2024-03-08T08:30:13.771384+0000","last_peered":"2024-03-08T08:30:13.771384+0000","last_clean":"2024-03-08T08:30:13.771384+0000","last_became_active":"2024-03-08T08:30:11.768616+0000","last_became_peered":"2024-03-08T08:30:11.768616+0000","last_unstale":"2024-03-08T08:30:13.771384+0000","last_undegraded":"2024-03-08T08:30:13.771384+0000","last_fullsized":"2024-03-08T08:30:13.771384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:37:34.473279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.789270+0000","last_change":"2024-03-08T08:30:11.776348+0000","last_active":"2024-03-08T08:30:12.789270+0000","last_peered":"2024-03-08T08:30:12.789270+0000","last_clean":"2024-03-08T08:30:12.789270+0000","last_became_active":"2024-03-08T08:30:11.776038+0000","last_became_peered":"2024-03-08T08:30:11.776038+0000","last_unstale":"2024-03-08T08:30:12.789270+0000","last_undegraded":"2024-03-08T08:30:12.789270+0000","last_fullsized":"2024-03-08T08:30:12.789270+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:36:28.049340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771737+0000","last_change":"2024-03-08T08:30:12.771667+0000","last_active":"2024-03-08T08:30:13.771737+0000","last_peered":"2024-03-08T08:30:13.771737+0000","last_clean":"2024-03-08T08:30:13.771737+0000","last_became_active":"2024-03-08T08:30:12.771019+0000","last_became_peered":"2024-03-08T08:30:12.771019+0000","last_unstale":"2024-03-08T08:30:13.771737+0000","last_undegraded":"2024-03-08T08:30:13.771737+0000","last_fullsized":"2024-03-08T08:30:13.771737+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:00:39.426592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771223+0000","last_change":"2024-03-08T08:30:11.768221+0000","last_active":"2024-03-08T08:30:13.771223+0000","last_peered":"2024-03-08T08:30:13.771223+0000","last_clean":"2024-03-08T08:30:13.771223+0000","last_became_active":"2024-03-08T08:30:11.767888+0000","last_became_peered":"2024-03-08T08:30:11.767888+0000","last_unstale":"2024-03-08T08:30:13.771223+0000","last_undegraded":"2024-03-08T08:30:13.771223+0000","last_fullsized":"2024-03-08T08:30:13.771223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:49.371937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771128+0000","last_change":"2024-03-08T08:30:12.778071+0000","last_active":"2024-03-08T08:30:13.771128+0000","last_peered":"2024-03-08T08:30:13.771128+0000","last_clean":"2024-03-08T08:30:13.771128+0000","last_became_active":"2024-03-08T08:30:12.777758+0000","last_became_peered":"2024-03-08T08:30:12.777758+0000","last_unstale":"2024-03-08T08:30:13.771128+0000","last_undegraded":"2024-03-08T08:30:13.771128+0000","last_fullsized":"2024-03-08T08:30:13.771128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:55.160157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.757891+0000","last_change":"2024-03-08T08:30:11.765902+0000","last_active":"2024-03-08T08:30:12.757891+0000","last_peered":"2024-03-08T08:30:12.757891+0000","last_clean":"2024-03-08T08:30:12.757891+0000","last_became_active":"2024-03-08T08:30:11.765630+0000","last_became_peered":"2024-03-08T08:30:11.765630+0000","last_unstale":"2024-03-08T08:30:12.757891+0000","last_undegraded":"2024-03-08T08:30:12.757891+0000","last_fullsized":"2024-03-08T08:30:12.757891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:31.519508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.779452+0000","last_change":"2024-03-08T08:30:12.776422+0000","last_active":"2024-03-08T08:30:12.779452+0000","last_peered":"2024-03-08T08:30:12.779452+0000","last_clean":"2024-03-08T08:30:12.779452+0000","last_became_active":"2024-03-08T08:30:12.776138+0000","last_became_peered":"2024-03-08T08:30:12.776138+0000","last_unstale":"2024-03-08T08:30:12.779452+0000","last_undegraded":"2024-03-08T08:30:12.779452+0000","last_fullsized":"2024-03-08T08:30:12.779452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771832+0000","last_change":"2024-03-08T08:30:11.765883+0000","last_active":"2024-03-08T08:30:13.771832+0000","last_peered":"2024-03-08T08:30:13.771832+0000","last_clean":"2024-03-08T08:30:13.771832+0000","last_became_active":"2024-03-08T08:30:11.765610+0000","last_became_peered":"2024-03-08T08:30:11.765610+0000","last_unstale":"2024-03-08T08:30:13.771832+0000","last_undegraded":"2024-03-08T08:30:13.771832+0000","last_fullsized":"2024-03-08T08:30:13.771832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:10:46.070078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771721+0000","last_change":"2024-03-08T08:30:12.766273+0000","last_active":"2024-03-08T08:30:13.771721+0000","last_peered":"2024-03-08T08:30:13.771721+0000","last_clean":"2024-03-08T08:30:13.771721+0000","last_became_active":"2024-03-08T08:30:12.765978+0000","last_became_peered":"2024-03-08T08:30:12.765978+0000","last_unstale":"2024-03-08T08:30:13.771721+0000","last_undegraded":"2024-03-08T08:30:13.771721+0000","last_fullsized":"2024-03-08T08:30:13.771721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:30:55.604670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.763669+0000","last_change":"2024-03-08T08:30:11.774576+0000","last_active":"2024-03-08T08:30:12.763669+0000","last_peered":"2024-03-08T08:30:12.763669+0000","last_clean":"2024-03-08T08:30:12.763669+0000","last_became_active":"2024-03-08T08:30:11.774252+0000","last_became_peered":"2024-03-08T08:30:11.774252+0000","last_unstale":"2024-03-08T08:30:12.763669+0000","last_undegraded":"2024-03-08T08:30:12.763669+0000","last_fullsized":"2024-03-08T08:30:12.763669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:05:06.478265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771172+0000","last_change":"2024-03-08T08:30:12.769391+0000","last_active":"2024-03-08T08:30:12.771172+0000","last_peered":"2024-03-08T08:30:12.771172+0000","last_clean":"2024-03-08T08:30:12.771172+0000","last_became_active":"2024-03-08T08:30:12.768633+0000","last_became_peered":"2024-03-08T08:30:12.768633+0000","last_unstale":"2024-03-08T08:30:12.771172+0000","last_undegraded":"2024-03-08T08:30:12.771172+0000","last_fullsized":"2024-03-08T08:30:12.771172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:15.001228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:10.743286+0000","last_change":"2024-03-08T08:30:10.743286+0000","last_active":"2024-03-08T08:30:10.743286+0000","last_peered":"2024-03-08T08:30:10.743286+0000","last_clean":"2024-03-08T08:30:10.743286+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:10.743286+0000","last_undegraded":"2024-03-08T08:30:10.743286+0000","last_fullsized":"2024-03-08T08:30:10.743286+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-08T08:30:11.745574+0000","last_change":"2024-03-08T08:30:11.745574+0000","last_active":"2024-03-08T08:30:11.745574+0000","last_peered":"2024-03-08T08:30:11.745574+0000","last_clean":"2024-03-08T08:30:11.745574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-08T08:30:11.745574+0000","last_undegraded":"2024-03-08T08:30:11.745574+0000","last_fullsized":"2024-03-08T08:30:11.745574+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770811+0000","last_change":"2024-03-08T08:30:11.764545+0000","last_active":"2024-03-08T08:30:13.770811+0000","last_peered":"2024-03-08T08:30:13.770811+0000","last_clean":"2024-03-08T08:30:13.770811+0000","last_became_active":"2024-03-08T08:30:11.764223+0000","last_became_peered":"2024-03-08T08:30:11.764223+0000","last_unstale":"2024-03-08T08:30:13.770811+0000","last_undegraded":"2024-03-08T08:30:13.770811+0000","last_fullsized":"2024-03-08T08:30:13.770811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:55:25.641965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771407+0000","last_change":"2024-03-08T08:30:12.778686+0000","last_active":"2024-03-08T08:30:13.771407+0000","last_peered":"2024-03-08T08:30:13.771407+0000","last_clean":"2024-03-08T08:30:13.771407+0000","last_became_active":"2024-03-08T08:30:12.778421+0000","last_became_peered":"2024-03-08T08:30:12.778421+0000","last_unstale":"2024-03-08T08:30:13.771407+0000","last_undegraded":"2024-03-08T08:30:13.771407+0000","last_fullsized":"2024-03-08T08:30:13.771407+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:36.852386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.783846+0000","last_change":"2024-03-08T08:30:11.763838+0000","last_active":"2024-03-08T08:30:12.783846+0000","last_peered":"2024-03-08T08:30:12.783846+0000","last_clean":"2024-03-08T08:30:12.783846+0000","last_became_active":"2024-03-08T08:30:11.763551+0000","last_became_peered":"2024-03-08T08:30:11.763551+0000","last_unstale":"2024-03-08T08:30:12.783846+0000","last_undegraded":"2024-03-08T08:30:12.783846+0000","last_fullsized":"2024-03-08T08:30:12.783846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:54.157869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771155+0000","last_change":"2024-03-08T08:30:12.780918+0000","last_active":"2024-03-08T08:30:13.771155+0000","last_peered":"2024-03-08T08:30:13.771155+0000","last_clean":"2024-03-08T08:30:13.771155+0000","last_became_active":"2024-03-08T08:30:12.780632+0000","last_became_peered":"2024-03-08T08:30:12.780632+0000","last_unstale":"2024-03-08T08:30:13.771155+0000","last_undegraded":"2024-03-08T08:30:13.771155+0000","last_fullsized":"2024-03-08T08:30:13.771155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:36:33.012077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771885+0000","last_change":"2024-03-08T08:30:11.771156+0000","last_active":"2024-03-08T08:30:13.771885+0000","last_peered":"2024-03-08T08:30:13.771885+0000","last_clean":"2024-03-08T08:30:13.771885+0000","last_became_active":"2024-03-08T08:30:11.770727+0000","last_became_peered":"2024-03-08T08:30:11.770727+0000","last_unstale":"2024-03-08T08:30:13.771885+0000","last_undegraded":"2024-03-08T08:30:13.771885+0000","last_fullsized":"2024-03-08T08:30:13.771885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:54:56.798285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771319+0000","last_change":"2024-03-08T08:30:12.765348+0000","last_active":"2024-03-08T08:30:13.771319+0000","last_peered":"2024-03-08T08:30:13.771319+0000","last_clean":"2024-03-08T08:30:13.771319+0000","last_became_active":"2024-03-08T08:30:12.765060+0000","last_became_peered":"2024-03-08T08:30:12.765060+0000","last_unstale":"2024-03-08T08:30:13.771319+0000","last_undegraded":"2024-03-08T08:30:13.771319+0000","last_fullsized":"2024-03-08T08:30:13.771319+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:46.691888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764732+0000","last_change":"2024-03-08T08:30:11.766097+0000","last_active":"2024-03-08T08:30:12.764732+0000","last_peered":"2024-03-08T08:30:12.764732+0000","last_clean":"2024-03-08T08:30:12.764732+0000","last_became_active":"2024-03-08T08:30:11.765725+0000","last_became_peered":"2024-03-08T08:30:11.765725+0000","last_unstale":"2024-03-08T08:30:12.764732+0000","last_undegraded":"2024-03-08T08:30:12.764732+0000","last_fullsized":"2024-03-08T08:30:12.764732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:10.853548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.770186+0000","last_change":"2024-03-08T08:30:12.765538+0000","last_active":"2024-03-08T08:30:12.770186+0000","last_peered":"2024-03-08T08:30:12.770186+0000","last_clean":"2024-03-08T08:30:12.770186+0000","last_became_active":"2024-03-08T08:30:12.765244+0000","last_became_peered":"2024-03-08T08:30:12.765244+0000","last_unstale":"2024-03-08T08:30:12.770186+0000","last_undegraded":"2024-03-08T08:30:12.770186+0000","last_fullsized":"2024-03-08T08:30:12.770186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764428+0000","last_change":"2024-03-08T08:30:11.765562+0000","last_active":"2024-03-08T08:30:12.764428+0000","last_peered":"2024-03-08T08:30:12.764428+0000","last_clean":"2024-03-08T08:30:12.764428+0000","last_became_active":"2024-03-08T08:30:11.765224+0000","last_became_peered":"2024-03-08T08:30:11.765224+0000","last_unstale":"2024-03-08T08:30:12.764428+0000","last_undegraded":"2024-03-08T08:30:12.764428+0000","last_fullsized":"2024-03-08T08:30:12.764428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:09.767738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781141+0000","last_change":"2024-03-08T08:30:12.769032+0000","last_active":"2024-03-08T08:30:12.781141+0000","last_peered":"2024-03-08T08:30:12.781141+0000","last_clean":"2024-03-08T08:30:12.781141+0000","last_became_active":"2024-03-08T08:30:12.768757+0000","last_became_peered":"2024-03-08T08:30:12.768757+0000","last_unstale":"2024-03-08T08:30:12.781141+0000","last_undegraded":"2024-03-08T08:30:12.781141+0000","last_fullsized":"2024-03-08T08:30:12.781141+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764030+0000","last_change":"2024-03-08T08:30:11.766780+0000","last_active":"2024-03-08T08:30:12.764030+0000","last_peered":"2024-03-08T08:30:12.764030+0000","last_clean":"2024-03-08T08:30:12.764030+0000","last_became_active":"2024-03-08T08:30:11.766314+0000","last_became_peered":"2024-03-08T08:30:11.766314+0000","last_unstale":"2024-03-08T08:30:12.764030+0000","last_undegraded":"2024-03-08T08:30:12.764030+0000","last_fullsized":"2024-03-08T08:30:12.764030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:45.689600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782871+0000","last_change":"2024-03-08T08:30:12.780285+0000","last_active":"2024-03-08T08:30:12.782871+0000","last_peered":"2024-03-08T08:30:12.782871+0000","last_clean":"2024-03-08T08:30:12.782871+0000","last_became_active":"2024-03-08T08:30:12.780006+0000","last_became_peered":"2024-03-08T08:30:12.780006+0000","last_unstale":"2024-03-08T08:30:12.782871+0000","last_undegraded":"2024-03-08T08:30:12.782871+0000","last_fullsized":"2024-03-08T08:30:12.782871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:55:26.644252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.762495+0000","last_change":"2024-03-08T08:30:11.775009+0000","last_active":"2024-03-08T08:30:12.762495+0000","last_peered":"2024-03-08T08:30:12.762495+0000","last_clean":"2024-03-08T08:30:12.762495+0000","last_became_active":"2024-03-08T08:30:11.774392+0000","last_became_peered":"2024-03-08T08:30:11.774392+0000","last_unstale":"2024-03-08T08:30:12.762495+0000","last_undegraded":"2024-03-08T08:30:12.762495+0000","last_fullsized":"2024-03-08T08:30:12.762495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:18:05.773445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770896+0000","last_change":"2024-03-08T08:30:12.782222+0000","last_active":"2024-03-08T08:30:13.770896+0000","last_peered":"2024-03-08T08:30:13.770896+0000","last_clean":"2024-03-08T08:30:13.770896+0000","last_became_active":"2024-03-08T08:30:12.781834+0000","last_became_peered":"2024-03-08T08:30:12.781834+0000","last_unstale":"2024-03-08T08:30:13.770896+0000","last_undegraded":"2024-03-08T08:30:13.770896+0000","last_fullsized":"2024-03-08T08:30:13.770896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:44:14.238079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.760309+0000","last_change":"2024-03-08T08:30:11.770901+0000","last_active":"2024-03-08T08:30:12.760309+0000","last_peered":"2024-03-08T08:30:12.760309+0000","last_clean":"2024-03-08T08:30:12.760309+0000","last_became_active":"2024-03-08T08:30:11.770626+0000","last_became_peered":"2024-03-08T08:30:11.770626+0000","last_unstale":"2024-03-08T08:30:12.760309+0000","last_undegraded":"2024-03-08T08:30:12.760309+0000","last_fullsized":"2024-03-08T08:30:12.760309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:25.994671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.769695+0000","last_change":"2024-03-08T08:30:12.765363+0000","last_active":"2024-03-08T08:30:12.769695+0000","last_peered":"2024-03-08T08:30:12.769695+0000","last_clean":"2024-03-08T08:30:12.769695+0000","last_became_active":"2024-03-08T08:30:12.765039+0000","last_became_peered":"2024-03-08T08:30:12.765039+0000","last_unstale":"2024-03-08T08:30:12.769695+0000","last_undegraded":"2024-03-08T08:30:12.769695+0000","last_fullsized":"2024-03-08T08:30:12.769695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771206+0000","last_change":"2024-03-08T08:30:11.767544+0000","last_active":"2024-03-08T08:30:13.771206+0000","last_peered":"2024-03-08T08:30:13.771206+0000","last_clean":"2024-03-08T08:30:13.771206+0000","last_became_active":"2024-03-08T08:30:11.766882+0000","last_became_peered":"2024-03-08T08:30:11.766882+0000","last_unstale":"2024-03-08T08:30:13.771206+0000","last_undegraded":"2024-03-08T08:30:13.771206+0000","last_fullsized":"2024-03-08T08:30:13.771206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:58:44.924554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":187,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-08T08:30:06.502346+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:30:06.502346+0000","last_peered":"2024-03-08T08:30:06.502346+0000","last_clean":"2024-03-08T08:30:06.502346+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:30:06.502346+0000","last_undegraded":"2024-03-08T08:30:06.502346+0000","last_fullsized":"2024-03-08T08:30:06.502346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":62,"acting":62,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":719587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":148108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":393216,"data_stored":149698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335540,"statfs":{"total":96636764160,"available":96599592960,"internally_reserved":0,"allocated":978944,"data_stored":739036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542162,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":397312,"data_stored":149720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":148862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:30:16.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:30:16.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.899+0000 7fb9859e9640 1 Processor -- start 2024-03-08T08:30:16.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.899+0000 7fb9859e9640 1 -- start start 2024-03-08T08:30:16.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb9859e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9800b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb9859e9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98015e1a0 con 0x7fb98015ddc0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb9859e9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800b8790 con 0x7fb9800d31c0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb9859e9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800b9030 con 0x7fb9800d1110 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb98015ddc0 0x7fb9800d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48174/0 (socket says 172.21.15.12:48174) 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97effd640 1 -- 172.21.15.12:0/2215485925 learned_addr learned my addr 172.21.15.12:0/2215485925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb977fff640 1 --2- 172.21.15.12:0/2215485925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9800b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140705896 0 0) 0x7fb98015e1a0 con 0x7fb98015ddc0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c003520 con 0x7fb98015ddc0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560015697 0 0) 0x7fb9800b8790 con 0x7fb9800d31c0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98015e1a0 con 0x7fb9800d31c0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278225240 0 0) 0x7fb96c003520 con 0x7fb98015ddc0 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb977fff640 1 -- 172.21.15.12:0/2215485925 >> v1:172.21.15.156:6789/0 conn(0x7fb9800d31c0 legacy=0x7fb9800b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb977fff640 1 -- 172.21.15.12:0/2215485925 >> v1:172.21.15.12:6789/0 conn(0x7fb98015ddc0 legacy=0x7fb9800d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.900+0000 7fb977fff640 1 -- 172.21.15.12:0/2215485925 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800b8a70 con 0x7fb9800d1110 2024-03-08T08:30:16.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.901+0000 7fb977fff640 1 --2- 172.21.15.12:0/2215485925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9800b6be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9680097d0 tx=0x7fb96802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5a38afbd178625 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.901+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968037070 con 0x7fb9800d1110 2024-03-08T08:30:16.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.901+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb968030510 con 0x7fb9800d1110 2024-03-08T08:30:16.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.901+0000 7fb97e7fc640 1 -- 172.21.15.12:0/2215485925 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970001670 con 0x7fb9800d1110 2024-03-08T08:30:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 -- 172.21.15.12:0/2215485925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 msgr2=0x7fb9800b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 --2- 172.21.15.12:0/2215485925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9800b6be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9680097d0 tx=0x7fb96802b800 comp rx=0 tx=0).stop 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 -- 172.21.15.12:0/2215485925 shutdown_connections 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 --2- 172.21.15.12:0/2215485925 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9800b6be0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 -- 172.21.15.12:0/2215485925 >> 172.21.15.12:0/2215485925 conn(0x7fb980072720 msgr2=0x7fb980072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 -- 172.21.15.12:0/2215485925 shutdown_connections 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.902+0000 7fb9859e9640 1 -- 172.21.15.12:0/2215485925 wait complete. 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.903+0000 7fb9859e9640 1 Processor -- start 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.903+0000 7fb9859e9640 1 -- start start 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9801f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb98015ddc0 0x7fb9800c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800b9980 con 0x7fb9800d31c0 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800d1d20 con 0x7fb98015ddc0 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb9859e9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800b7830 con 0x7fb9800d1110 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50148/0 (socket says 172.21.15.12:50148) 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 -- 172.21.15.12:0/1860854027 learned_addr learned my addr 172.21.15.12:0/1860854027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb97f7fe640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb98015ddc0 0x7fb9800c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb97effd640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9801f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 -- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 msgr2=0x7fb9801f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9801f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 -- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb98015ddc0 msgr2=0x7fb9800c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb98015ddc0 0x7fb9800c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 -- 172.21.15.12:0/1860854027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9801f7ad0 con 0x7fb9800d31c0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb97effd640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9801f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.904+0000 7fb977fff640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb968000c00 tx=0x7fb9680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.905+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968037040 con 0x7fb9800d31c0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.905+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb968004250 con 0x7fb9800d31c0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.905+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb980160900 con 0x7fb9800d31c0 2024-03-08T08:30:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.905+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96803c670 con 0x7fb9800d31c0 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.905+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980160ec0 con 0x7fb9800d31c0 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.906+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9800749c0 con 0x7fb9800d31c0 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.906+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb96803a070 con 0x7fb9800d31c0 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.906+0000 7fb97cff9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 0x7fb96006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.906+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9680abc10 con 0x7fb9800d31c0 2024-03-08T08:30:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.906+0000 7fb97effd640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 0x7fb96006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.907+0000 7fb97effd640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 0x7fb96006d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9700017e0 tx=0x7fb970009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:16.911+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb968041020 con 0x7fb9800d31c0 2024-03-08T08:30:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.086+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb98015d330 con 0x7fb96006b4d0 2024-03-08T08:30:17.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.092+0000 7fb97cff9640 1 -- 172.21.15.12:0/1860854027 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251912 (secure 0 0 0) 0x7fb98015d330 con 0x7fb96006b4d0 2024-03-08T08:30:17.096 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:17.097 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 msgr2=0x7fb96006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 0x7fb96006d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9700017e0 tx=0x7fb970009040 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 msgr2=0x7fb980156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb968000c00 tx=0x7fb9680046c0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 shutdown_connections 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb96006b4d0 0x7fb96006d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb98015ddc0 0x7fb9800c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9800d31c0 0x7fb980156300 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 --2- 172.21.15.12:0/1860854027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9800d1110 0x7fb9801f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 >> 172.21.15.12:0/1860854027 conn(0x7fb980072720 msgr2=0x7fb9800b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:17.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 shutdown_connections 2024-03-08T08:30:17.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.095+0000 7fb9859e9640 1 -- 172.21.15.12:0/1860854027 wait complete. 2024-03-08T08:30:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-08T08:30:17.007002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241176,"kb_used_data":4212,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733544,"statfs":{"total":773094113280,"available":772847149056,"internally_reserved":0,"allocated":4313088,"data_stored":2376544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001070"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781694+0000","last_change":"2024-03-08T08:30:12.771963+0000","last_active":"2024-03-08T08:30:12.781694+0000","last_peered":"2024-03-08T08:30:12.781694+0000","last_clean":"2024-03-08T08:30:12.781694+0000","last_became_active":"2024-03-08T08:30:12.771583+0000","last_became_peered":"2024-03-08T08:30:12.771583+0000","last_unstale":"2024-03-08T08:30:12.781694+0000","last_undegraded":"2024-03-08T08:30:12.781694+0000","last_fullsized":"2024-03-08T08:30:12.781694+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:10.770026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771813+0000","last_change":"2024-03-08T08:30:12.768931+0000","last_active":"2024-03-08T08:30:13.771813+0000","last_peered":"2024-03-08T08:30:13.771813+0000","last_clean":"2024-03-08T08:30:13.771813+0000","last_became_active":"2024-03-08T08:30:12.768657+0000","last_became_peered":"2024-03-08T08:30:12.768657+0000","last_unstale":"2024-03-08T08:30:13.771813+0000","last_undegraded":"2024-03-08T08:30:13.771813+0000","last_fullsized":"2024-03-08T08:30:13.771813+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:13:03.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414740+0000","last_change":"2024-03-08T08:30:12.785429+0000","last_active":"2024-03-08T08:30:14.414740+0000","last_peered":"2024-03-08T08:30:14.414740+0000","last_clean":"2024-03-08T08:30:14.414740+0000","last_became_active":"2024-03-08T08:30:12.785125+0000","last_became_peered":"2024-03-08T08:30:12.785125+0000","last_unstale":"2024-03-08T08:30:14.414740+0000","last_undegraded":"2024-03-08T08:30:14.414740+0000","last_fullsized":"2024-03-08T08:30:14.414740+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:00:16.041830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784141+0000","last_change":"2024-03-08T08:30:12.779832+0000","last_active":"2024-03-08T08:30:12.784141+0000","last_peered":"2024-03-08T08:30:12.784141+0000","last_clean":"2024-03-08T08:30:12.784141+0000","last_became_active":"2024-03-08T08:30:12.779569+0000","last_became_peered":"2024-03-08T08:30:12.779569+0000","last_unstale":"2024-03-08T08:30:12.784141+0000","last_undegraded":"2024-03-08T08:30:12.784141+0000","last_fullsized":"2024-03-08T08:30:12.784141+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:22:19.728810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771709+0000","last_change":"2024-03-08T08:30:12.773763+0000","last_active":"2024-03-08T08:30:13.771709+0000","last_peered":"2024-03-08T08:30:13.771709+0000","last_clean":"2024-03-08T08:30:13.771709+0000","last_became_active":"2024-03-08T08:30:12.773441+0000","last_became_peered":"2024-03-08T08:30:12.773441+0000","last_unstale":"2024-03-08T08:30:13.771709+0000","last_undegraded":"2024-03-08T08:30:13.771709+0000","last_fullsized":"2024-03-08T08:30:13.771709+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:52:59.982314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771054+0000","last_change":"2024-03-08T08:30:12.774949+0000","last_active":"2024-03-08T08:30:13.771054+0000","last_peered":"2024-03-08T08:30:13.771054+0000","last_clean":"2024-03-08T08:30:13.771054+0000","last_became_active":"2024-03-08T08:30:12.774445+0000","last_became_peered":"2024-03-08T08:30:12.774445+0000","last_unstale":"2024-03-08T08:30:13.771054+0000","last_undegraded":"2024-03-08T08:30:13.771054+0000","last_fullsized":"2024-03-08T08:30:13.771054+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:39:22.720405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784128+0000","last_change":"2024-03-08T08:30:12.778655+0000","last_active":"2024-03-08T08:30:12.784128+0000","last_peered":"2024-03-08T08:30:12.784128+0000","last_clean":"2024-03-08T08:30:12.784128+0000","last_became_active":"2024-03-08T08:30:12.778365+0000","last_became_peered":"2024-03-08T08:30:12.778365+0000","last_unstale":"2024-03-08T08:30:12.784128+0000","last_undegraded":"2024-03-08T08:30:12.784128+0000","last_fullsized":"2024-03-08T08:30:12.784128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784061+0000","last_change":"2024-03-08T08:30:12.773939+0000","last_active":"2024-03-08T08:30:12.784061+0000","last_peered":"2024-03-08T08:30:12.784061+0000","last_clean":"2024-03-08T08:30:12.784061+0000","last_became_active":"2024-03-08T08:30:12.773643+0000","last_became_peered":"2024-03-08T08:30:12.773643+0000","last_unstale":"2024-03-08T08:30:12.784061+0000","last_undegraded":"2024-03-08T08:30:12.784061+0000","last_fullsized":"2024-03-08T08:30:12.784061+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:11.855836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414659+0000","last_change":"2024-03-08T08:30:12.784022+0000","last_active":"2024-03-08T08:30:14.414659+0000","last_peered":"2024-03-08T08:30:14.414659+0000","last_clean":"2024-03-08T08:30:14.414659+0000","last_became_active":"2024-03-08T08:30:12.783730+0000","last_became_peered":"2024-03-08T08:30:12.783730+0000","last_unstale":"2024-03-08T08:30:14.414659+0000","last_undegraded":"2024-03-08T08:30:14.414659+0000","last_fullsized":"2024-03-08T08:30:14.414659+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:53:16.744830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.783164+0000","last_change":"2024-03-08T08:30:12.778247+0000","last_active":"2024-03-08T08:30:12.783164+0000","last_peered":"2024-03-08T08:30:12.783164+0000","last_clean":"2024-03-08T08:30:12.783164+0000","last_became_active":"2024-03-08T08:30:12.777949+0000","last_became_peered":"2024-03-08T08:30:12.777949+0000","last_unstale":"2024-03-08T08:30:12.783164+0000","last_undegraded":"2024-03-08T08:30:12.783164+0000","last_fullsized":"2024-03-08T08:30:12.783164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:18.773299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771128+0000","last_change":"2024-03-08T08:30:12.771143+0000","last_active":"2024-03-08T08:30:13.771128+0000","last_peered":"2024-03-08T08:30:13.771128+0000","last_clean":"2024-03-08T08:30:13.771128+0000","last_became_active":"2024-03-08T08:30:12.770848+0000","last_became_peered":"2024-03-08T08:30:12.770848+0000","last_unstale":"2024-03-08T08:30:13.771128+0000","last_undegraded":"2024-03-08T08:30:13.771128+0000","last_fullsized":"2024-03-08T08:30:13.771128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:23:01.198137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414791+0000","last_change":"2024-03-08T08:30:12.781133+0000","last_active":"2024-03-08T08:30:14.414791+0000","last_peered":"2024-03-08T08:30:14.414791+0000","last_clean":"2024-03-08T08:30:14.414791+0000","last_became_active":"2024-03-08T08:30:12.780872+0000","last_became_peered":"2024-03-08T08:30:12.780872+0000","last_unstale":"2024-03-08T08:30:14.414791+0000","last_undegraded":"2024-03-08T08:30:14.414791+0000","last_fullsized":"2024-03-08T08:30:14.414791+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:50:11.734921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771142+0000","last_change":"2024-03-08T08:30:12.769530+0000","last_active":"2024-03-08T08:30:12.771142+0000","last_peered":"2024-03-08T08:30:12.771142+0000","last_clean":"2024-03-08T08:30:12.771142+0000","last_became_active":"2024-03-08T08:30:12.769004+0000","last_became_peered":"2024-03-08T08:30:12.769004+0000","last_unstale":"2024-03-08T08:30:12.771142+0000","last_undegraded":"2024-03-08T08:30:12.771142+0000","last_fullsized":"2024-03-08T08:30:12.771142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781794+0000","last_change":"2024-03-08T08:30:12.771997+0000","last_active":"2024-03-08T08:30:12.781794+0000","last_peered":"2024-03-08T08:30:12.781794+0000","last_clean":"2024-03-08T08:30:12.781794+0000","last_became_active":"2024-03-08T08:30:12.771384+0000","last_became_peered":"2024-03-08T08:30:12.771384+0000","last_unstale":"2024-03-08T08:30:12.781794+0000","last_undegraded":"2024-03-08T08:30:12.781794+0000","last_fullsized":"2024-03-08T08:30:12.781794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.777421+0000","last_change":"2024-03-08T08:30:12.772353+0000","last_active":"2024-03-08T08:30:12.777421+0000","last_peered":"2024-03-08T08:30:12.777421+0000","last_clean":"2024-03-08T08:30:12.777421+0000","last_became_active":"2024-03-08T08:30:12.772053+0000","last_became_peered":"2024-03-08T08:30:12.772053+0000","last_unstale":"2024-03-08T08:30:12.777421+0000","last_undegraded":"2024-03-08T08:30:12.777421+0000","last_fullsized":"2024-03-08T08:30:12.777421+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:17:12.613908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764859+0000","last_change":"2024-03-08T08:30:12.762997+0000","last_active":"2024-03-08T08:30:12.764859+0000","last_peered":"2024-03-08T08:30:12.764859+0000","last_clean":"2024-03-08T08:30:12.764859+0000","last_became_active":"2024-03-08T08:30:12.762670+0000","last_became_peered":"2024-03-08T08:30:12.762670+0000","last_unstale":"2024-03-08T08:30:12.764859+0000","last_undegraded":"2024-03-08T08:30:12.764859+0000","last_fullsized":"2024-03-08T08:30:12.764859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771114+0000","last_change":"2024-03-08T08:30:12.786769+0000","last_active":"2024-03-08T08:30:13.771114+0000","last_peered":"2024-03-08T08:30:13.771114+0000","last_clean":"2024-03-08T08:30:13.771114+0000","last_became_active":"2024-03-08T08:30:12.786508+0000","last_became_peered":"2024-03-08T08:30:12.786508+0000","last_unstale":"2024-03-08T08:30:13.771114+0000","last_undegraded":"2024-03-08T08:30:13.771114+0000","last_fullsized":"2024-03-08T08:30:13.771114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:46:59.687919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771338+0000","last_change":"2024-03-08T08:30:12.770360+0000","last_active":"2024-03-08T08:30:13.771338+0000","last_peered":"2024-03-08T08:30:13.771338+0000","last_clean":"2024-03-08T08:30:13.771338+0000","last_became_active":"2024-03-08T08:30:12.770023+0000","last_became_peered":"2024-03-08T08:30:12.770023+0000","last_unstale":"2024-03-08T08:30:13.771338+0000","last_undegraded":"2024-03-08T08:30:13.771338+0000","last_fullsized":"2024-03-08T08:30:13.771338+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:10.770026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771312+0000","last_change":"2024-03-08T08:30:12.772061+0000","last_active":"2024-03-08T08:30:13.771312+0000","last_peered":"2024-03-08T08:30:13.771312+0000","last_clean":"2024-03-08T08:30:13.771312+0000","last_became_active":"2024-03-08T08:30:12.771756+0000","last_became_peered":"2024-03-08T08:30:12.771756+0000","last_unstale":"2024-03-08T08:30:13.771312+0000","last_undegraded":"2024-03-08T08:30:13.771312+0000","last_fullsized":"2024-03-08T08:30:13.771312+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:05:07.480553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.772721+0000","last_change":"2024-03-08T08:30:12.769148+0000","last_active":"2024-03-08T08:30:12.772721+0000","last_peered":"2024-03-08T08:30:12.772721+0000","last_clean":"2024-03-08T08:30:12.772721+0000","last_became_active":"2024-03-08T08:30:12.768864+0000","last_became_peered":"2024-03-08T08:30:12.768864+0000","last_unstale":"2024-03-08T08:30:12.772721+0000","last_undegraded":"2024-03-08T08:30:12.772721+0000","last_fullsized":"2024-03-08T08:30:12.772721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:26.996959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.778836+0000","last_change":"2024-03-08T08:30:12.773425+0000","last_active":"2024-03-08T08:30:12.778836+0000","last_peered":"2024-03-08T08:30:12.778836+0000","last_clean":"2024-03-08T08:30:12.778836+0000","last_became_active":"2024-03-08T08:30:12.772885+0000","last_became_peered":"2024-03-08T08:30:12.772885+0000","last_unstale":"2024-03-08T08:30:12.778836+0000","last_undegraded":"2024-03-08T08:30:12.778836+0000","last_fullsized":"2024-03-08T08:30:12.778836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:53:18.773299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788520+0000","last_change":"2024-03-08T08:30:12.785148+0000","last_active":"2024-03-08T08:30:12.788520+0000","last_peered":"2024-03-08T08:30:12.788520+0000","last_clean":"2024-03-08T08:30:12.788520+0000","last_became_active":"2024-03-08T08:30:12.784849+0000","last_became_peered":"2024-03-08T08:30:12.784849+0000","last_unstale":"2024-03-08T08:30:12.788520+0000","last_undegraded":"2024-03-08T08:30:12.788520+0000","last_fullsized":"2024-03-08T08:30:12.788520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781030+0000","last_change":"2024-03-08T08:30:12.779346+0000","last_active":"2024-03-08T08:30:12.781030+0000","last_peered":"2024-03-08T08:30:12.781030+0000","last_clean":"2024-03-08T08:30:12.781030+0000","last_became_active":"2024-03-08T08:30:12.778776+0000","last_became_peered":"2024-03-08T08:30:12.778776+0000","last_unstale":"2024-03-08T08:30:12.781030+0000","last_undegraded":"2024-03-08T08:30:12.781030+0000","last_fullsized":"2024-03-08T08:30:12.781030+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784094+0000","last_change":"2024-03-08T08:30:12.774559+0000","last_active":"2024-03-08T08:30:12.784094+0000","last_peered":"2024-03-08T08:30:12.784094+0000","last_clean":"2024-03-08T08:30:12.784094+0000","last_became_active":"2024-03-08T08:30:12.774290+0000","last_became_peered":"2024-03-08T08:30:12.774290+0000","last_unstale":"2024-03-08T08:30:12.784094+0000","last_undegraded":"2024-03-08T08:30:12.784094+0000","last_fullsized":"2024-03-08T08:30:12.784094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:15.001228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782740+0000","last_change":"2024-03-08T08:30:12.776173+0000","last_active":"2024-03-08T08:30:12.782740+0000","last_peered":"2024-03-08T08:30:12.782740+0000","last_clean":"2024-03-08T08:30:12.782740+0000","last_became_active":"2024-03-08T08:30:12.775895+0000","last_became_peered":"2024-03-08T08:30:12.775895+0000","last_unstale":"2024-03-08T08:30:12.782740+0000","last_undegraded":"2024-03-08T08:30:12.782740+0000","last_fullsized":"2024-03-08T08:30:12.782740+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:17:12.613908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784503+0000","last_change":"2024-03-08T08:30:12.783252+0000","last_active":"2024-03-08T08:30:12.784503+0000","last_peered":"2024-03-08T08:30:12.784503+0000","last_clean":"2024-03-08T08:30:12.784503+0000","last_became_active":"2024-03-08T08:30:12.782968+0000","last_became_peered":"2024-03-08T08:30:12.782968+0000","last_unstale":"2024-03-08T08:30:12.784503+0000","last_undegraded":"2024-03-08T08:30:12.784503+0000","last_fullsized":"2024-03-08T08:30:12.784503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:22:36.472741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771644+0000","last_change":"2024-03-08T08:30:12.767160+0000","last_active":"2024-03-08T08:30:13.771644+0000","last_peered":"2024-03-08T08:30:13.771644+0000","last_clean":"2024-03-08T08:30:13.771644+0000","last_became_active":"2024-03-08T08:30:12.766783+0000","last_became_peered":"2024-03-08T08:30:12.766783+0000","last_unstale":"2024-03-08T08:30:13.771644+0000","last_undegraded":"2024-03-08T08:30:13.771644+0000","last_fullsized":"2024-03-08T08:30:13.771644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:03:00.953042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784245+0000","last_change":"2024-03-08T08:30:12.774300+0000","last_active":"2024-03-08T08:30:12.784245+0000","last_peered":"2024-03-08T08:30:12.784245+0000","last_clean":"2024-03-08T08:30:12.784245+0000","last_became_active":"2024-03-08T08:30:12.773829+0000","last_became_peered":"2024-03-08T08:30:12.773829+0000","last_unstale":"2024-03-08T08:30:12.784245+0000","last_undegraded":"2024-03-08T08:30:12.784245+0000","last_fullsized":"2024-03-08T08:30:12.784245+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:26.996959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.778812+0000","last_change":"2024-03-08T08:30:12.773181+0000","last_active":"2024-03-08T08:30:12.778812+0000","last_peered":"2024-03-08T08:30:12.778812+0000","last_clean":"2024-03-08T08:30:12.778812+0000","last_became_active":"2024-03-08T08:30:12.772567+0000","last_became_peered":"2024-03-08T08:30:12.772567+0000","last_unstale":"2024-03-08T08:30:12.778812+0000","last_undegraded":"2024-03-08T08:30:12.778812+0000","last_fullsized":"2024-03-08T08:30:12.778812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:13:35.419025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771213+0000","last_change":"2024-03-08T08:30:12.770045+0000","last_active":"2024-03-08T08:30:12.771213+0000","last_peered":"2024-03-08T08:30:12.771213+0000","last_clean":"2024-03-08T08:30:12.771213+0000","last_became_active":"2024-03-08T08:30:12.769707+0000","last_became_peered":"2024-03-08T08:30:12.769707+0000","last_unstale":"2024-03-08T08:30:12.771213+0000","last_undegraded":"2024-03-08T08:30:12.771213+0000","last_fullsized":"2024-03-08T08:30:12.771213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781201+0000","last_change":"2024-03-08T08:30:12.773265+0000","last_active":"2024-03-08T08:30:12.781201+0000","last_peered":"2024-03-08T08:30:12.781201+0000","last_clean":"2024-03-08T08:30:12.781201+0000","last_became_active":"2024-03-08T08:30:12.772979+0000","last_became_peered":"2024-03-08T08:30:12.772979+0000","last_unstale":"2024-03-08T08:30:12.781201+0000","last_undegraded":"2024-03-08T08:30:12.781201+0000","last_fullsized":"2024-03-08T08:30:12.781201+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:36.852386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770760+0000","last_change":"2024-03-08T08:30:12.774695+0000","last_active":"2024-03-08T08:30:13.770760+0000","last_peered":"2024-03-08T08:30:13.770760+0000","last_clean":"2024-03-08T08:30:13.770760+0000","last_became_active":"2024-03-08T08:30:12.774417+0000","last_became_peered":"2024-03-08T08:30:12.774417+0000","last_unstale":"2024-03-08T08:30:13.770760+0000","last_undegraded":"2024-03-08T08:30:13.770760+0000","last_fullsized":"2024-03-08T08:30:13.770760+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:37:31.325473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.780665+0000","last_change":"2024-03-08T08:30:12.778985+0000","last_active":"2024-03-08T08:30:12.780665+0000","last_peered":"2024-03-08T08:30:12.780665+0000","last_clean":"2024-03-08T08:30:12.780665+0000","last_became_active":"2024-03-08T08:30:12.778722+0000","last_became_peered":"2024-03-08T08:30:12.778722+0000","last_unstale":"2024-03-08T08:30:12.780665+0000","last_undegraded":"2024-03-08T08:30:12.780665+0000","last_fullsized":"2024-03-08T08:30:12.780665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771228+0000","last_change":"2024-03-08T08:30:11.772402+0000","last_active":"2024-03-08T08:30:13.771228+0000","last_peered":"2024-03-08T08:30:13.771228+0000","last_clean":"2024-03-08T08:30:13.771228+0000","last_became_active":"2024-03-08T08:30:11.772090+0000","last_became_peered":"2024-03-08T08:30:11.772090+0000","last_unstale":"2024-03-08T08:30:13.771228+0000","last_undegraded":"2024-03-08T08:30:13.771228+0000","last_fullsized":"2024-03-08T08:30:13.771228+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:22:35.470453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781668+0000","last_change":"2024-03-08T08:30:12.771686+0000","last_active":"2024-03-08T08:30:12.781668+0000","last_peered":"2024-03-08T08:30:12.781668+0000","last_clean":"2024-03-08T08:30:12.781668+0000","last_became_active":"2024-03-08T08:30:12.770935+0000","last_became_peered":"2024-03-08T08:30:12.770935+0000","last_unstale":"2024-03-08T08:30:12.781668+0000","last_undegraded":"2024-03-08T08:30:12.781668+0000","last_fullsized":"2024-03-08T08:30:12.781668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:46.691888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.786866+0000","last_change":"2024-03-08T08:30:11.768998+0000","last_active":"2024-03-08T08:30:12.786866+0000","last_peered":"2024-03-08T08:30:12.786866+0000","last_clean":"2024-03-08T08:30:12.786866+0000","last_became_active":"2024-03-08T08:30:11.768654+0000","last_became_peered":"2024-03-08T08:30:11.768654+0000","last_unstale":"2024-03-08T08:30:12.786866+0000","last_undegraded":"2024-03-08T08:30:12.786866+0000","last_fullsized":"2024-03-08T08:30:12.786866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:10.853548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784265+0000","last_change":"2024-03-08T08:30:12.774724+0000","last_active":"2024-03-08T08:30:12.784265+0000","last_peered":"2024-03-08T08:30:12.784265+0000","last_clean":"2024-03-08T08:30:12.784265+0000","last_became_active":"2024-03-08T08:30:12.774439+0000","last_became_peered":"2024-03-08T08:30:12.774439+0000","last_unstale":"2024-03-08T08:30:12.784265+0000","last_undegraded":"2024-03-08T08:30:12.784265+0000","last_fullsized":"2024-03-08T08:30:12.784265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771440+0000","last_change":"2024-03-08T08:30:11.772633+0000","last_active":"2024-03-08T08:30:13.771440+0000","last_peered":"2024-03-08T08:30:13.771440+0000","last_clean":"2024-03-08T08:30:13.771440+0000","last_became_active":"2024-03-08T08:30:11.772209+0000","last_became_peered":"2024-03-08T08:30:11.772209+0000","last_unstale":"2024-03-08T08:30:13.771440+0000","last_undegraded":"2024-03-08T08:30:13.771440+0000","last_fullsized":"2024-03-08T08:30:13.771440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:06:20.197407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781393+0000","last_change":"2024-03-08T08:30:12.773677+0000","last_active":"2024-03-08T08:30:12.781393+0000","last_peered":"2024-03-08T08:30:12.781393+0000","last_clean":"2024-03-08T08:30:12.781393+0000","last_became_active":"2024-03-08T08:30:12.773407+0000","last_became_peered":"2024-03-08T08:30:12.773407+0000","last_unstale":"2024-03-08T08:30:12.781393+0000","last_undegraded":"2024-03-08T08:30:12.781393+0000","last_fullsized":"2024-03-08T08:30:12.781393+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:42:12.720015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.785451+0000","last_change":"2024-03-08T08:30:11.768037+0000","last_active":"2024-03-08T08:30:12.785451+0000","last_peered":"2024-03-08T08:30:12.785451+0000","last_clean":"2024-03-08T08:30:12.785451+0000","last_became_active":"2024-03-08T08:30:11.767329+0000","last_became_peered":"2024-03-08T08:30:11.767329+0000","last_unstale":"2024-03-08T08:30:12.785451+0000","last_undegraded":"2024-03-08T08:30:12.785451+0000","last_fullsized":"2024-03-08T08:30:12.785451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:13.998940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414617+0000","last_change":"2024-03-08T08:30:12.784387+0000","last_active":"2024-03-08T08:30:14.414617+0000","last_peered":"2024-03-08T08:30:14.414617+0000","last_clean":"2024-03-08T08:30:14.414617+0000","last_became_active":"2024-03-08T08:30:12.783689+0000","last_became_peered":"2024-03-08T08:30:12.783689+0000","last_unstale":"2024-03-08T08:30:14.414617+0000","last_undegraded":"2024-03-08T08:30:14.414617+0000","last_fullsized":"2024-03-08T08:30:14.414617+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:12:31.109574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788898+0000","last_change":"2024-03-08T08:30:11.772419+0000","last_active":"2024-03-08T08:30:12.788898+0000","last_peered":"2024-03-08T08:30:12.788898+0000","last_clean":"2024-03-08T08:30:12.788898+0000","last_became_active":"2024-03-08T08:30:11.772166+0000","last_became_peered":"2024-03-08T08:30:11.772166+0000","last_unstale":"2024-03-08T08:30:12.788898+0000","last_undegraded":"2024-03-08T08:30:12.788898+0000","last_fullsized":"2024-03-08T08:30:12.788898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:39:21.718117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784179+0000","last_change":"2024-03-08T08:30:12.779459+0000","last_active":"2024-03-08T08:30:12.784179+0000","last_peered":"2024-03-08T08:30:12.784179+0000","last_clean":"2024-03-08T08:30:12.784179+0000","last_became_active":"2024-03-08T08:30:12.778200+0000","last_became_peered":"2024-03-08T08:30:12.778200+0000","last_unstale":"2024-03-08T08:30:12.784179+0000","last_undegraded":"2024-03-08T08:30:12.784179+0000","last_fullsized":"2024-03-08T08:30:12.784179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.788556+0000","last_change":"2024-03-08T08:30:11.770944+0000","last_active":"2024-03-08T08:30:12.788556+0000","last_peered":"2024-03-08T08:30:12.788556+0000","last_clean":"2024-03-08T08:30:12.788556+0000","last_became_active":"2024-03-08T08:30:11.770228+0000","last_became_peered":"2024-03-08T08:30:11.770228+0000","last_unstale":"2024-03-08T08:30:12.788556+0000","last_undegraded":"2024-03-08T08:30:12.788556+0000","last_fullsized":"2024-03-08T08:30:12.788556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:29:57.683266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414777+0000","last_change":"2024-03-08T08:30:12.780159+0000","last_active":"2024-03-08T08:30:14.414777+0000","last_peered":"2024-03-08T08:30:14.414777+0000","last_clean":"2024-03-08T08:30:14.414777+0000","last_became_active":"2024-03-08T08:30:12.779809+0000","last_became_peered":"2024-03-08T08:30:12.779809+0000","last_unstale":"2024-03-08T08:30:14.414777+0000","last_undegraded":"2024-03-08T08:30:14.414777+0000","last_fullsized":"2024-03-08T08:30:14.414777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:17:50.841720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770987+0000","last_change":"2024-03-08T08:30:11.762487+0000","last_active":"2024-03-08T08:30:13.770987+0000","last_peered":"2024-03-08T08:30:13.770987+0000","last_clean":"2024-03-08T08:30:13.770987+0000","last_became_active":"2024-03-08T08:30:11.762176+0000","last_became_peered":"2024-03-08T08:30:11.762176+0000","last_unstale":"2024-03-08T08:30:13.770987+0000","last_undegraded":"2024-03-08T08:30:13.770987+0000","last_fullsized":"2024-03-08T08:30:13.770987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:03:13.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781342+0000","last_change":"2024-03-08T08:30:12.770530+0000","last_active":"2024-03-08T08:30:12.781342+0000","last_peered":"2024-03-08T08:30:12.781342+0000","last_clean":"2024-03-08T08:30:12.781342+0000","last_became_active":"2024-03-08T08:30:12.770235+0000","last_became_peered":"2024-03-08T08:30:12.770235+0000","last_unstale":"2024-03-08T08:30:12.781342+0000","last_undegraded":"2024-03-08T08:30:12.781342+0000","last_fullsized":"2024-03-08T08:30:12.781342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.763191+0000","last_change":"2024-03-08T08:30:11.775419+0000","last_active":"2024-03-08T08:30:12.763191+0000","last_peered":"2024-03-08T08:30:12.763191+0000","last_clean":"2024-03-08T08:30:12.763191+0000","last_became_active":"2024-03-08T08:30:11.775074+0000","last_became_peered":"2024-03-08T08:30:11.775074+0000","last_unstale":"2024-03-08T08:30:12.763191+0000","last_undegraded":"2024-03-08T08:30:12.763191+0000","last_fullsized":"2024-03-08T08:30:12.763191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:49:55.509413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771049+0000","last_change":"2024-03-08T08:30:12.773585+0000","last_active":"2024-03-08T08:30:13.771049+0000","last_peered":"2024-03-08T08:30:13.771049+0000","last_clean":"2024-03-08T08:30:13.771049+0000","last_became_active":"2024-03-08T08:30:12.773306+0000","last_became_peered":"2024-03-08T08:30:12.773306+0000","last_unstale":"2024-03-08T08:30:13.771049+0000","last_undegraded":"2024-03-08T08:30:13.771049+0000","last_fullsized":"2024-03-08T08:30:13.771049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:07:40.421935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.772067+0000","last_change":"2024-03-08T08:30:11.770207+0000","last_active":"2024-03-08T08:30:13.772067+0000","last_peered":"2024-03-08T08:30:13.772067+0000","last_clean":"2024-03-08T08:30:13.772067+0000","last_became_active":"2024-03-08T08:30:11.769935+0000","last_became_peered":"2024-03-08T08:30:11.769935+0000","last_unstale":"2024-03-08T08:30:13.772067+0000","last_undegraded":"2024-03-08T08:30:13.772067+0000","last_fullsized":"2024-03-08T08:30:13.772067+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T15:35:56.043443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781076+0000","last_change":"2024-03-08T08:30:12.769250+0000","last_active":"2024-03-08T08:30:12.781076+0000","last_peered":"2024-03-08T08:30:12.781076+0000","last_clean":"2024-03-08T08:30:12.781076+0000","last_became_active":"2024-03-08T08:30:12.768740+0000","last_became_peered":"2024-03-08T08:30:12.768740+0000","last_unstale":"2024-03-08T08:30:12.781076+0000","last_undegraded":"2024-03-08T08:30:12.781076+0000","last_fullsized":"2024-03-08T08:30:12.781076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771770+0000","last_change":"2024-03-08T08:30:11.774201+0000","last_active":"2024-03-08T08:30:13.771770+0000","last_peered":"2024-03-08T08:30:13.771770+0000","last_clean":"2024-03-08T08:30:13.771770+0000","last_became_active":"2024-03-08T08:30:11.773794+0000","last_became_peered":"2024-03-08T08:30:11.773794+0000","last_unstale":"2024-03-08T08:30:13.771770+0000","last_undegraded":"2024-03-08T08:30:13.771770+0000","last_fullsized":"2024-03-08T08:30:13.771770+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:11:09.060145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784101+0000","last_change":"2024-03-08T08:30:12.778402+0000","last_active":"2024-03-08T08:30:12.784101+0000","last_peered":"2024-03-08T08:30:12.784101+0000","last_clean":"2024-03-08T08:30:12.784101+0000","last_became_active":"2024-03-08T08:30:12.777928+0000","last_became_peered":"2024-03-08T08:30:12.777928+0000","last_unstale":"2024-03-08T08:30:12.784101+0000","last_undegraded":"2024-03-08T08:30:12.784101+0000","last_fullsized":"2024-03-08T08:30:12.784101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:49:34.108392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771100+0000","last_change":"2024-03-08T08:30:11.767315+0000","last_active":"2024-03-08T08:30:13.771100+0000","last_peered":"2024-03-08T08:30:13.771100+0000","last_clean":"2024-03-08T08:30:13.771100+0000","last_became_active":"2024-03-08T08:30:11.766482+0000","last_became_peered":"2024-03-08T08:30:11.766482+0000","last_unstale":"2024-03-08T08:30:13.771100+0000","last_undegraded":"2024-03-08T08:30:13.771100+0000","last_fullsized":"2024-03-08T08:30:13.771100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:40:43.309762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770951+0000","last_change":"2024-03-08T08:30:12.782524+0000","last_active":"2024-03-08T08:30:13.770951+0000","last_peered":"2024-03-08T08:30:13.770951+0000","last_clean":"2024-03-08T08:30:13.770951+0000","last_became_active":"2024-03-08T08:30:12.782202+0000","last_became_peered":"2024-03-08T08:30:12.782202+0000","last_unstale":"2024-03-08T08:30:13.770951+0000","last_undegraded":"2024-03-08T08:30:13.770951+0000","last_fullsized":"2024-03-08T08:30:13.770951+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:05:56.432265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.790230+0000","last_change":"2024-03-08T08:30:11.768457+0000","last_active":"2024-03-08T08:30:12.790230+0000","last_peered":"2024-03-08T08:30:12.790230+0000","last_clean":"2024-03-08T08:30:12.790230+0000","last_became_active":"2024-03-08T08:30:11.768076+0000","last_became_peered":"2024-03-08T08:30:11.768076+0000","last_unstale":"2024-03-08T08:30:12.790230+0000","last_undegraded":"2024-03-08T08:30:12.790230+0000","last_fullsized":"2024-03-08T08:30:12.790230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:35.850098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.779430+0000","last_change":"2024-03-08T08:30:12.772512+0000","last_active":"2024-03-08T08:30:12.779430+0000","last_peered":"2024-03-08T08:30:12.779430+0000","last_clean":"2024-03-08T08:30:12.779430+0000","last_became_active":"2024-03-08T08:30:12.772000+0000","last_became_peered":"2024-03-08T08:30:12.772000+0000","last_unstale":"2024-03-08T08:30:12.779430+0000","last_undegraded":"2024-03-08T08:30:12.779430+0000","last_fullsized":"2024-03-08T08:30:12.779430+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:19:15.268206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.789527+0000","last_change":"2024-03-08T08:30:11.774795+0000","last_active":"2024-03-08T08:30:12.789527+0000","last_peered":"2024-03-08T08:30:12.789527+0000","last_clean":"2024-03-08T08:30:12.789527+0000","last_became_active":"2024-03-08T08:30:11.773890+0000","last_became_peered":"2024-03-08T08:30:11.773890+0000","last_unstale":"2024-03-08T08:30:12.789527+0000","last_undegraded":"2024-03-08T08:30:12.789527+0000","last_fullsized":"2024-03-08T08:30:12.789527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:54.157869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782955+0000","last_change":"2024-03-08T08:30:12.777692+0000","last_active":"2024-03-08T08:30:12.782955+0000","last_peered":"2024-03-08T08:30:12.782955+0000","last_clean":"2024-03-08T08:30:12.782955+0000","last_became_active":"2024-03-08T08:30:12.775558+0000","last_became_peered":"2024-03-08T08:30:12.775558+0000","last_unstale":"2024-03-08T08:30:12.782955+0000","last_undegraded":"2024-03-08T08:30:12.782955+0000","last_fullsized":"2024-03-08T08:30:12.782955+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:13:35.419025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.758388+0000","last_change":"2024-03-08T08:30:11.766601+0000","last_active":"2024-03-08T08:30:12.758388+0000","last_peered":"2024-03-08T08:30:12.758388+0000","last_clean":"2024-03-08T08:30:12.758388+0000","last_became_active":"2024-03-08T08:30:11.766308+0000","last_became_peered":"2024-03-08T08:30:11.766308+0000","last_unstale":"2024-03-08T08:30:12.758388+0000","last_undegraded":"2024-03-08T08:30:12.758388+0000","last_fullsized":"2024-03-08T08:30:12.758388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:35.850098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.784305+0000","last_change":"2024-03-08T08:30:12.779637+0000","last_active":"2024-03-08T08:30:12.784305+0000","last_peered":"2024-03-08T08:30:12.784305+0000","last_clean":"2024-03-08T08:30:12.784305+0000","last_became_active":"2024-03-08T08:30:12.779156+0000","last_became_peered":"2024-03-08T08:30:12.779156+0000","last_unstale":"2024-03-08T08:30:12.784305+0000","last_undegraded":"2024-03-08T08:30:12.784305+0000","last_fullsized":"2024-03-08T08:30:12.784305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.757723+0000","last_change":"2024-03-08T08:30:11.768109+0000","last_active":"2024-03-08T08:30:12.757723+0000","last_peered":"2024-03-08T08:30:12.757723+0000","last_clean":"2024-03-08T08:30:12.757723+0000","last_became_active":"2024-03-08T08:30:11.767831+0000","last_became_peered":"2024-03-08T08:30:11.767831+0000","last_unstale":"2024-03-08T08:30:12.757723+0000","last_undegraded":"2024-03-08T08:30:12.757723+0000","last_fullsized":"2024-03-08T08:30:12.757723+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:09.767738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770861+0000","last_change":"2024-03-08T08:30:12.786123+0000","last_active":"2024-03-08T08:30:13.770861+0000","last_peered":"2024-03-08T08:30:13.770861+0000","last_clean":"2024-03-08T08:30:13.770861+0000","last_became_active":"2024-03-08T08:30:12.785834+0000","last_became_peered":"2024-03-08T08:30:12.785834+0000","last_unstale":"2024-03-08T08:30:13.770861+0000","last_undegraded":"2024-03-08T08:30:13.770861+0000","last_fullsized":"2024-03-08T08:30:13.770861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:41:40.311673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770896+0000","last_change":"2024-03-08T08:30:11.766823+0000","last_active":"2024-03-08T08:30:13.770896+0000","last_peered":"2024-03-08T08:30:13.770896+0000","last_clean":"2024-03-08T08:30:13.770896+0000","last_became_active":"2024-03-08T08:30:11.766490+0000","last_became_peered":"2024-03-08T08:30:11.766490+0000","last_unstale":"2024-03-08T08:30:13.770896+0000","last_undegraded":"2024-03-08T08:30:13.770896+0000","last_fullsized":"2024-03-08T08:30:13.770896+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:48:35.326526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771101+0000","last_change":"2024-03-08T08:30:12.770278+0000","last_active":"2024-03-08T08:30:13.771101+0000","last_peered":"2024-03-08T08:30:13.771101+0000","last_clean":"2024-03-08T08:30:13.771101+0000","last_became_active":"2024-03-08T08:30:12.769978+0000","last_became_peered":"2024-03-08T08:30:12.769978+0000","last_unstale":"2024-03-08T08:30:13.771101+0000","last_undegraded":"2024-03-08T08:30:13.771101+0000","last_fullsized":"2024-03-08T08:30:13.771101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:47:29.489464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771384+0000","last_change":"2024-03-08T08:30:11.769883+0000","last_active":"2024-03-08T08:30:13.771384+0000","last_peered":"2024-03-08T08:30:13.771384+0000","last_clean":"2024-03-08T08:30:13.771384+0000","last_became_active":"2024-03-08T08:30:11.768616+0000","last_became_peered":"2024-03-08T08:30:11.768616+0000","last_unstale":"2024-03-08T08:30:13.771384+0000","last_undegraded":"2024-03-08T08:30:13.771384+0000","last_fullsized":"2024-03-08T08:30:13.771384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:37:34.473279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414864+0000","last_change":"2024-03-08T08:30:12.780709+0000","last_active":"2024-03-08T08:30:14.414864+0000","last_peered":"2024-03-08T08:30:14.414864+0000","last_clean":"2024-03-08T08:30:14.414864+0000","last_became_active":"2024-03-08T08:30:12.780418+0000","last_became_peered":"2024-03-08T08:30:12.780418+0000","last_unstale":"2024-03-08T08:30:14.414864+0000","last_undegraded":"2024-03-08T08:30:14.414864+0000","last_fullsized":"2024-03-08T08:30:14.414864+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:21.207273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.789270+0000","last_change":"2024-03-08T08:30:11.776348+0000","last_active":"2024-03-08T08:30:12.789270+0000","last_peered":"2024-03-08T08:30:12.789270+0000","last_clean":"2024-03-08T08:30:12.789270+0000","last_became_active":"2024-03-08T08:30:11.776038+0000","last_became_peered":"2024-03-08T08:30:11.776038+0000","last_unstale":"2024-03-08T08:30:12.789270+0000","last_undegraded":"2024-03-08T08:30:12.789270+0000","last_fullsized":"2024-03-08T08:30:12.789270+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:36:28.049340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771737+0000","last_change":"2024-03-08T08:30:12.771667+0000","last_active":"2024-03-08T08:30:13.771737+0000","last_peered":"2024-03-08T08:30:13.771737+0000","last_clean":"2024-03-08T08:30:13.771737+0000","last_became_active":"2024-03-08T08:30:12.771019+0000","last_became_peered":"2024-03-08T08:30:12.771019+0000","last_unstale":"2024-03-08T08:30:13.771737+0000","last_undegraded":"2024-03-08T08:30:13.771737+0000","last_fullsized":"2024-03-08T08:30:13.771737+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:00:39.426592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771223+0000","last_change":"2024-03-08T08:30:11.768221+0000","last_active":"2024-03-08T08:30:13.771223+0000","last_peered":"2024-03-08T08:30:13.771223+0000","last_clean":"2024-03-08T08:30:13.771223+0000","last_became_active":"2024-03-08T08:30:11.767888+0000","last_became_peered":"2024-03-08T08:30:11.767888+0000","last_unstale":"2024-03-08T08:30:13.771223+0000","last_undegraded":"2024-03-08T08:30:13.771223+0000","last_fullsized":"2024-03-08T08:30:13.771223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:02:49.371937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771128+0000","last_change":"2024-03-08T08:30:12.778071+0000","last_active":"2024-03-08T08:30:13.771128+0000","last_peered":"2024-03-08T08:30:13.771128+0000","last_clean":"2024-03-08T08:30:13.771128+0000","last_became_active":"2024-03-08T08:30:12.777758+0000","last_became_peered":"2024-03-08T08:30:12.777758+0000","last_unstale":"2024-03-08T08:30:13.771128+0000","last_undegraded":"2024-03-08T08:30:13.771128+0000","last_fullsized":"2024-03-08T08:30:13.771128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:55.160157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.757891+0000","last_change":"2024-03-08T08:30:11.765902+0000","last_active":"2024-03-08T08:30:12.757891+0000","last_peered":"2024-03-08T08:30:12.757891+0000","last_clean":"2024-03-08T08:30:12.757891+0000","last_became_active":"2024-03-08T08:30:11.765630+0000","last_became_peered":"2024-03-08T08:30:11.765630+0000","last_unstale":"2024-03-08T08:30:12.757891+0000","last_undegraded":"2024-03-08T08:30:12.757891+0000","last_fullsized":"2024-03-08T08:30:12.757891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:31.519508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.779452+0000","last_change":"2024-03-08T08:30:12.776422+0000","last_active":"2024-03-08T08:30:12.779452+0000","last_peered":"2024-03-08T08:30:12.779452+0000","last_clean":"2024-03-08T08:30:12.779452+0000","last_became_active":"2024-03-08T08:30:12.776138+0000","last_became_peered":"2024-03-08T08:30:12.776138+0000","last_unstale":"2024-03-08T08:30:12.779452+0000","last_undegraded":"2024-03-08T08:30:12.779452+0000","last_fullsized":"2024-03-08T08:30:12.779452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771832+0000","last_change":"2024-03-08T08:30:11.765883+0000","last_active":"2024-03-08T08:30:13.771832+0000","last_peered":"2024-03-08T08:30:13.771832+0000","last_clean":"2024-03-08T08:30:13.771832+0000","last_became_active":"2024-03-08T08:30:11.765610+0000","last_became_peered":"2024-03-08T08:30:11.765610+0000","last_unstale":"2024-03-08T08:30:13.771832+0000","last_undegraded":"2024-03-08T08:30:13.771832+0000","last_fullsized":"2024-03-08T08:30:13.771832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:10:46.070078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771721+0000","last_change":"2024-03-08T08:30:12.766273+0000","last_active":"2024-03-08T08:30:13.771721+0000","last_peered":"2024-03-08T08:30:13.771721+0000","last_clean":"2024-03-08T08:30:13.771721+0000","last_became_active":"2024-03-08T08:30:12.765978+0000","last_became_peered":"2024-03-08T08:30:12.765978+0000","last_unstale":"2024-03-08T08:30:13.771721+0000","last_undegraded":"2024-03-08T08:30:13.771721+0000","last_fullsized":"2024-03-08T08:30:13.771721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:30:55.604670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.763669+0000","last_change":"2024-03-08T08:30:11.774576+0000","last_active":"2024-03-08T08:30:12.763669+0000","last_peered":"2024-03-08T08:30:12.763669+0000","last_clean":"2024-03-08T08:30:12.763669+0000","last_became_active":"2024-03-08T08:30:11.774252+0000","last_became_peered":"2024-03-08T08:30:11.774252+0000","last_unstale":"2024-03-08T08:30:12.763669+0000","last_undegraded":"2024-03-08T08:30:12.763669+0000","last_fullsized":"2024-03-08T08:30:12.763669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:05:06.478265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.771172+0000","last_change":"2024-03-08T08:30:12.769391+0000","last_active":"2024-03-08T08:30:12.771172+0000","last_peered":"2024-03-08T08:30:12.771172+0000","last_clean":"2024-03-08T08:30:12.771172+0000","last_became_active":"2024-03-08T08:30:12.768633+0000","last_became_peered":"2024-03-08T08:30:12.768633+0000","last_unstale":"2024-03-08T08:30:12.771172+0000","last_undegraded":"2024-03-08T08:30:12.771172+0000","last_fullsized":"2024-03-08T08:30:12.771172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:11:15.001228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414442+0000","last_change":"2024-03-08T08:30:11.773485+0000","last_active":"2024-03-08T08:30:14.414442+0000","last_peered":"2024-03-08T08:30:14.414442+0000","last_clean":"2024-03-08T08:30:14.414442+0000","last_became_active":"2024-03-08T08:30:11.773090+0000","last_became_peered":"2024-03-08T08:30:11.773090+0000","last_unstale":"2024-03-08T08:30:14.414442+0000","last_undegraded":"2024-03-08T08:30:14.414442+0000","last_fullsized":"2024-03-08T08:30:14.414442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:20:21.837771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414511+0000","last_change":"2024-03-08T08:30:12.782690+0000","last_active":"2024-03-08T08:30:14.414511+0000","last_peered":"2024-03-08T08:30:14.414511+0000","last_clean":"2024-03-08T08:30:14.414511+0000","last_became_active":"2024-03-08T08:30:12.782397+0000","last_became_peered":"2024-03-08T08:30:12.782397+0000","last_unstale":"2024-03-08T08:30:14.414511+0000","last_undegraded":"2024-03-08T08:30:14.414511+0000","last_fullsized":"2024-03-08T08:30:14.414511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:31:24.300727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770811+0000","last_change":"2024-03-08T08:30:11.764545+0000","last_active":"2024-03-08T08:30:13.770811+0000","last_peered":"2024-03-08T08:30:13.770811+0000","last_clean":"2024-03-08T08:30:13.770811+0000","last_became_active":"2024-03-08T08:30:11.764223+0000","last_became_peered":"2024-03-08T08:30:11.764223+0000","last_unstale":"2024-03-08T08:30:13.770811+0000","last_undegraded":"2024-03-08T08:30:13.770811+0000","last_fullsized":"2024-03-08T08:30:13.770811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:55:25.641965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771407+0000","last_change":"2024-03-08T08:30:12.778686+0000","last_active":"2024-03-08T08:30:13.771407+0000","last_peered":"2024-03-08T08:30:13.771407+0000","last_clean":"2024-03-08T08:30:13.771407+0000","last_became_active":"2024-03-08T08:30:12.778421+0000","last_became_peered":"2024-03-08T08:30:12.778421+0000","last_unstale":"2024-03-08T08:30:13.771407+0000","last_undegraded":"2024-03-08T08:30:13.771407+0000","last_fullsized":"2024-03-08T08:30:13.771407+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T16:58:36.852386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.783846+0000","last_change":"2024-03-08T08:30:11.763838+0000","last_active":"2024-03-08T08:30:12.783846+0000","last_peered":"2024-03-08T08:30:12.783846+0000","last_clean":"2024-03-08T08:30:12.783846+0000","last_became_active":"2024-03-08T08:30:11.763551+0000","last_became_peered":"2024-03-08T08:30:11.763551+0000","last_unstale":"2024-03-08T08:30:12.783846+0000","last_undegraded":"2024-03-08T08:30:12.783846+0000","last_fullsized":"2024-03-08T08:30:12.783846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T18:33:54.157869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771155+0000","last_change":"2024-03-08T08:30:12.780918+0000","last_active":"2024-03-08T08:30:13.771155+0000","last_peered":"2024-03-08T08:30:13.771155+0000","last_clean":"2024-03-08T08:30:13.771155+0000","last_became_active":"2024-03-08T08:30:12.780632+0000","last_became_peered":"2024-03-08T08:30:12.780632+0000","last_unstale":"2024-03-08T08:30:13.771155+0000","last_undegraded":"2024-03-08T08:30:13.771155+0000","last_fullsized":"2024-03-08T08:30:13.771155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:36:33.012077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771885+0000","last_change":"2024-03-08T08:30:11.771156+0000","last_active":"2024-03-08T08:30:13.771885+0000","last_peered":"2024-03-08T08:30:13.771885+0000","last_clean":"2024-03-08T08:30:13.771885+0000","last_became_active":"2024-03-08T08:30:11.770727+0000","last_became_peered":"2024-03-08T08:30:11.770727+0000","last_unstale":"2024-03-08T08:30:13.771885+0000","last_undegraded":"2024-03-08T08:30:13.771885+0000","last_fullsized":"2024-03-08T08:30:13.771885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T12:54:56.798285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771319+0000","last_change":"2024-03-08T08:30:12.765348+0000","last_active":"2024-03-08T08:30:13.771319+0000","last_peered":"2024-03-08T08:30:13.771319+0000","last_clean":"2024-03-08T08:30:13.771319+0000","last_became_active":"2024-03-08T08:30:12.765060+0000","last_became_peered":"2024-03-08T08:30:12.765060+0000","last_unstale":"2024-03-08T08:30:13.771319+0000","last_undegraded":"2024-03-08T08:30:13.771319+0000","last_fullsized":"2024-03-08T08:30:13.771319+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:46.691888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764732+0000","last_change":"2024-03-08T08:30:11.766097+0000","last_active":"2024-03-08T08:30:12.764732+0000","last_peered":"2024-03-08T08:30:12.764732+0000","last_clean":"2024-03-08T08:30:12.764732+0000","last_became_active":"2024-03-08T08:30:11.765725+0000","last_became_peered":"2024-03-08T08:30:11.765725+0000","last_unstale":"2024-03-08T08:30:12.764732+0000","last_undegraded":"2024-03-08T08:30:12.764732+0000","last_fullsized":"2024-03-08T08:30:12.764732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:56:10.853548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.770186+0000","last_change":"2024-03-08T08:30:12.765538+0000","last_active":"2024-03-08T08:30:12.770186+0000","last_peered":"2024-03-08T08:30:12.770186+0000","last_clean":"2024-03-08T08:30:12.770186+0000","last_became_active":"2024-03-08T08:30:12.765244+0000","last_became_peered":"2024-03-08T08:30:12.765244+0000","last_unstale":"2024-03-08T08:30:12.770186+0000","last_undegraded":"2024-03-08T08:30:12.770186+0000","last_fullsized":"2024-03-08T08:30:12.770186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:19:09.530477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764428+0000","last_change":"2024-03-08T08:30:11.765562+0000","last_active":"2024-03-08T08:30:12.764428+0000","last_peered":"2024-03-08T08:30:12.764428+0000","last_clean":"2024-03-08T08:30:12.764428+0000","last_became_active":"2024-03-08T08:30:11.765224+0000","last_became_peered":"2024-03-08T08:30:11.765224+0000","last_unstale":"2024-03-08T08:30:12.764428+0000","last_undegraded":"2024-03-08T08:30:12.764428+0000","last_fullsized":"2024-03-08T08:30:12.764428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T13:46:09.767738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.781141+0000","last_change":"2024-03-08T08:30:12.769032+0000","last_active":"2024-03-08T08:30:12.781141+0000","last_peered":"2024-03-08T08:30:12.781141+0000","last_clean":"2024-03-08T08:30:12.781141+0000","last_became_active":"2024-03-08T08:30:12.768757+0000","last_became_peered":"2024-03-08T08:30:12.768757+0000","last_unstale":"2024-03-08T08:30:12.781141+0000","last_undegraded":"2024-03-08T08:30:12.781141+0000","last_fullsized":"2024-03-08T08:30:12.781141+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T11:45:32.521796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.764030+0000","last_change":"2024-03-08T08:30:11.766780+0000","last_active":"2024-03-08T08:30:12.764030+0000","last_peered":"2024-03-08T08:30:12.764030+0000","last_clean":"2024-03-08T08:30:12.764030+0000","last_became_active":"2024-03-08T08:30:11.766314+0000","last_became_peered":"2024-03-08T08:30:11.766314+0000","last_unstale":"2024-03-08T08:30:12.764030+0000","last_undegraded":"2024-03-08T08:30:12.764030+0000","last_fullsized":"2024-03-08T08:30:12.764030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T20:09:45.689600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.782871+0000","last_change":"2024-03-08T08:30:12.780285+0000","last_active":"2024-03-08T08:30:12.782871+0000","last_peered":"2024-03-08T08:30:12.782871+0000","last_clean":"2024-03-08T08:30:12.782871+0000","last_became_active":"2024-03-08T08:30:12.780006+0000","last_became_peered":"2024-03-08T08:30:12.780006+0000","last_unstale":"2024-03-08T08:30:12.782871+0000","last_undegraded":"2024-03-08T08:30:12.782871+0000","last_fullsized":"2024-03-08T08:30:12.782871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:55:26.644252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.762495+0000","last_change":"2024-03-08T08:30:11.775009+0000","last_active":"2024-03-08T08:30:12.762495+0000","last_peered":"2024-03-08T08:30:12.762495+0000","last_clean":"2024-03-08T08:30:12.762495+0000","last_became_active":"2024-03-08T08:30:11.774392+0000","last_became_peered":"2024-03-08T08:30:11.774392+0000","last_unstale":"2024-03-08T08:30:12.762495+0000","last_undegraded":"2024-03-08T08:30:12.762495+0000","last_fullsized":"2024-03-08T08:30:12.762495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T17:18:05.773445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.770896+0000","last_change":"2024-03-08T08:30:12.782222+0000","last_active":"2024-03-08T08:30:13.770896+0000","last_peered":"2024-03-08T08:30:13.770896+0000","last_clean":"2024-03-08T08:30:13.770896+0000","last_became_active":"2024-03-08T08:30:12.781834+0000","last_became_peered":"2024-03-08T08:30:12.781834+0000","last_unstale":"2024-03-08T08:30:13.770896+0000","last_undegraded":"2024-03-08T08:30:13.770896+0000","last_fullsized":"2024-03-08T08:30:13.770896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T19:44:14.238079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.760309+0000","last_change":"2024-03-08T08:30:11.770901+0000","last_active":"2024-03-08T08:30:12.760309+0000","last_peered":"2024-03-08T08:30:12.760309+0000","last_clean":"2024-03-08T08:30:12.760309+0000","last_became_active":"2024-03-08T08:30:11.770626+0000","last_became_peered":"2024-03-08T08:30:11.770626+0000","last_unstale":"2024-03-08T08:30:12.760309+0000","last_undegraded":"2024-03-08T08:30:12.760309+0000","last_fullsized":"2024-03-08T08:30:12.760309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:33:25.994671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-08T08:30:12.769695+0000","last_change":"2024-03-08T08:30:12.765363+0000","last_active":"2024-03-08T08:30:12.769695+0000","last_peered":"2024-03-08T08:30:12.769695+0000","last_clean":"2024-03-08T08:30:12.769695+0000","last_became_active":"2024-03-08T08:30:12.765039+0000","last_became_peered":"2024-03-08T08:30:12.765039+0000","last_unstale":"2024-03-08T08:30:12.769695+0000","last_undegraded":"2024-03-08T08:30:12.769695+0000","last_fullsized":"2024-03-08T08:30:12.769695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:11.745574+0000","last_clean_scrub_stamp":"2024-03-08T08:30:11.745574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T09:58:33.948138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:13.771206+0000","last_change":"2024-03-08T08:30:11.767544+0000","last_active":"2024-03-08T08:30:13.771206+0000","last_peered":"2024-03-08T08:30:13.771206+0000","last_clean":"2024-03-08T08:30:13.771206+0000","last_became_active":"2024-03-08T08:30:11.766882+0000","last_became_peered":"2024-03-08T08:30:11.766882+0000","last_unstale":"2024-03-08T08:30:13.771206+0000","last_undegraded":"2024-03-08T08:30:13.771206+0000","last_fullsized":"2024-03-08T08:30:13.771206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:30:10.743286+0000","last_clean_scrub_stamp":"2024-03-08T08:30:10.743286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T08:58:44.924554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-08T08:30:14.414441+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:30:14.414441+0000","last_peered":"2024-03-08T08:30:14.414441+0000","last_clean":"2024-03-08T08:30:14.414441+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:30:14.414441+0000","last_undegraded":"2024-03-08T08:30:14.414441+0000","last_fullsized":"2024-03-08T08:30:14.414441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":991232,"data_stored":744824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":148108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":393216,"data_stored":149698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335540,"statfs":{"total":96636764160,"available":96599592960,"internally_reserved":0,"allocated":978944,"data_stored":739036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542162,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":397312,"data_stored":149720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":148862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:30:17.111 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-08T08:30:17.111 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-08T08:30:17.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.254+0000 7f754a44e640 1 Processor -- start 2024-03-08T08:30:17.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.254+0000 7f754a44e640 1 -- start start 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f754a44e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7544161260 0x7f75440b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f754a44e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440d1d90 con 0x7f75440b6b30 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f754a44e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440d2070 con 0x7f754406c870 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f754a44e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440d2350 con 0x7f7544161260 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f7543fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7544161260 0x7f75440b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f7543fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7544161260 0x7f75440b93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50150/0 (socket says 172.21.15.12:50150) 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f75437fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f75440b6b30 0x7f754406c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48176/0 (socket says 172.21.15.12:48176) 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f7543fff640 1 -- 172.21.15.12:0/3724451438 learned_addr learned my addr 172.21.15.12:0/3724451438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.255+0000 7f7543fff640 1 -- 172.21.15.12:0/3724451438 >> v1:172.21.15.156:6789/0 conn(0x7f754406c870 legacy=0x7f75440d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7543fff640 1 -- 172.21.15.12:0/3724451438 >> v1:172.21.15.12:6789/0 conn(0x7f75440b6b30 legacy=0x7f754406c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7543fff640 1 -- 172.21.15.12:0/3724451438 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440d2630 con 0x7f7544161260 2024-03-08T08:30:17.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7542ffd640 1 -- 172.21.15.12:0/3724451438 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598366616 0 0) 0x7f75440d1d90 con 0x7f75440b6b30 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7543fff640 1 --2- 172.21.15.12:0/3724451438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7544161260 0x7f75440b93b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7534009870 tx=0x7f753402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469de093338f7286 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7542ffd640 1 -- 172.21.15.12:0/3724451438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534034070 con 0x7f7544161260 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7542ffd640 1 -- 172.21.15.12:0/3724451438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7534030860 con 0x7f7544161260 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.256+0000 7f7542ffd640 1 -- 172.21.15.12:0/3724451438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534030c30 con 0x7f7544161260 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 -- 172.21.15.12:0/3724451438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7544161260 msgr2=0x7f75440b93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 --2- 172.21.15.12:0/3724451438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7544161260 0x7f75440b93b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7534009870 tx=0x7f753402b9a0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 -- 172.21.15.12:0/3724451438 shutdown_connections 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 --2- 172.21.15.12:0/3724451438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7544161260 0x7f75440b93b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 -- 172.21.15.12:0/3724451438 >> 172.21.15.12:0/3724451438 conn(0x7f7544072930 msgr2=0x7f75440d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 -- 172.21.15.12:0/3724451438 shutdown_connections 2024-03-08T08:30:17.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.257+0000 7f754a44e640 1 -- 172.21.15.12:0/3724451438 wait complete. 2024-03-08T08:30:17.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.258+0000 7f754a44e640 1 Processor -- start 2024-03-08T08:30:17.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.258+0000 7f754a44e640 1 -- start start 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 0x7f7544149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 0x7f75440c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75440b9a70 con 0x7f75440b6b30 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75440b98f0 con 0x7f7544161260 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f754a44e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75440d3540 con 0x7f754406c870 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50154/0 (socket says 172.21.15.12:50154) 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 -- 172.21.15.12:0/3237401383 learned_addr learned my addr 172.21.15.12:0/3237401383 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75489eb640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 0x7f75440c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f7543fff640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 0x7f7544149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 -- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 msgr2=0x7f7544149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 0x7f7544149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 -- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 msgr2=0x7f75440c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 0x7f75440c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75437fe640 1 -- 172.21.15.12:0/3237401383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754414b200 con 0x7f75440b6b30 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f75489eb640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 0x7f75440c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.259+0000 7f7543fff640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 0x7f7544149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f75437fe640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c4d0 tx=0x7f752c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c018070 con 0x7f75440b6b30 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c007500 con 0x7f75440b6b30 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754414b4e0 con 0x7f75440b6b30 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c004a20 con 0x7f75440b6b30 2024-03-08T08:30:17.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.260+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754414baa0 con 0x7f75440b6b30 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.261+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75440749c0 con 0x7f75440b6b30 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.261+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f752c01b070 con 0x7f75440b6b30 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.262+0000 7f7540ff9640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 0x7f752406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.262+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f752c089450 con 0x7f75440b6b30 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.262+0000 7f7543fff640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 0x7f752406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.262+0000 7f7543fff640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 0x7f752406d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7534036000 tx=0x7f753403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.266+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f752c058360 con 0x7f75440b6b30 2024-03-08T08:30:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.514+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f75440b35d0 con 0x7f75440b6b30 2024-03-08T08:30:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.516+0000 7f7540ff9640 1 -- 172.21.15.12:0/3237401383 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f752c057c60 con 0x7f75440b6b30 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAhzOplr8ebChAA8n4wUJZpCY0XA2qC5ELxpw== 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stdout: caps mds = "allow" 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stdout: caps mon = "allow r" 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-08T08:30:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 msgr2=0x7f752406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 0x7f752406d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7534036000 tx=0x7f753403b040 comp rx=0 tx=0).stop 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 msgr2=0x7f754414a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c4d0 tx=0x7f752c00c990 comp rx=0 tx=0).stop 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 shutdown_connections 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f752406b500 0x7f752406d9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7544161260 0x7f75440c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75440b6b30 0x7f754414a920 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 --2- 172.21.15.12:0/3237401383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f754406c870 0x7f7544149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.518+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 >> 172.21.15.12:0/3237401383 conn(0x7f7544072930 msgr2=0x7f75440b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.519+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 shutdown_connections 2024-03-08T08:30:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.519+0000 7f754a44e640 1 -- 172.21.15.12:0/3237401383 wait complete. 2024-03-08T08:30:17.531 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:30:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 Processor -- start 2024-03-08T08:30:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 -- start start 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb84815be40 0x7fb84809aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb84809af90 0x7fb84806f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480cd890 con 0x7fb848099ca0 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8480d09b0 con 0x7fb84809af90 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.630+0000 7fb84e2ae640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8480cc7f0 con 0x7fb84815be40 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb84809af90 0x7fb84806f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb84809af90 0x7fb84806f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50162/0 (socket says 172.21.15.12:50162) 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 -- 172.21.15.12:0/4086557769 learned_addr learned my addr 172.21.15.12:0/4086557769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb847fff640 1 --1- 172.21.15.12:0/4086557769 >> v1:172.21.15.12:6789/0 conn(0x7fb848099ca0 0x7fb84809a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48180/0 (socket says 172.21.15.12:48180) 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb8477fe640 1 --2- 172.21.15.12:0/4086557769 >> v2:172.21.15.156:3300/0 conn(0x7fb84815be40 0x7fb84809aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 -- 172.21.15.12:0/4086557769 >> v2:172.21.15.156:3300/0 conn(0x7fb84815be40 msgr2=0x7fb84809aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 --2- 172.21.15.12:0/4086557769 >> v2:172.21.15.156:3300/0 conn(0x7fb84815be40 0x7fb84809aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 -- 172.21.15.12:0/4086557769 >> v1:172.21.15.12:6789/0 conn(0x7fb848099ca0 legacy=0x7fb84809a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb84c84b640 1 -- 172.21.15.12:0/4086557769 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480ccad0 con 0x7fb84809af90 2024-03-08T08:30:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.631+0000 7fb846ffd640 1 -- 172.21.15.12:0/4086557769 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625173757 0 0) 0x7fb838001110 con 0x7fb848099ca0 2024-03-08T08:30:17.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84c84b640 1 --2- 172.21.15.12:0/4086557769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb84809af90 0x7fb84806f7e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8480d0ce0 tx=0x7fb84002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895f9cf1a2f54ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb846ffd640 1 -- 172.21.15.12:0/4086557769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84002cb20 con 0x7fb84809af90 2024-03-08T08:30:17.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb846ffd640 1 -- 172.21.15.12:0/4086557769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84002ccc0 con 0x7fb84809af90 2024-03-08T08:30:17.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb846ffd640 1 -- 172.21.15.12:0/4086557769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840034610 con 0x7fb84809af90 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 -- 172.21.15.12:0/4086557769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb84809af90 msgr2=0x7fb84806f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/4086557769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb84809af90 0x7fb84806f7e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8480d0ce0 tx=0x7fb84002bf80 comp rx=0 tx=0).stop 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 -- 172.21.15.12:0/4086557769 shutdown_connections 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/4086557769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb84809af90 0x7fb84806f7e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/4086557769 >> v2:172.21.15.156:3300/0 conn(0x7fb84815be40 0x7fb84809aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 -- 172.21.15.12:0/4086557769 >> 172.21.15.12:0/4086557769 conn(0x7fb8480d1900 msgr2=0x7fb8480d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 -- 172.21.15.12:0/4086557769 shutdown_connections 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.632+0000 7fb84e2ae640 1 -- 172.21.15.12:0/4086557769 wait complete. 2024-03-08T08:30:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 Processor -- start 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 -- start start 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb84809af90 0x7fb8480c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb84815be40 0x7fb8481f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480cc4a0 con 0x7fb848099ca0 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480cd090 con 0x7fb84809af90 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.634+0000 7fb84e2ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84806fd20 con 0x7fb84815be40 2024-03-08T08:30:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50168/0 (socket says 172.21.15.12:50168) 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 -- 172.21.15.12:0/3584190144 learned_addr learned my addr 172.21.15.12:0/3584190144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb8477fe640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb84809af90 0x7fb8480c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb84c84b640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb84815be40 0x7fb8481f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 -- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb84815be40 msgr2=0x7fb8481f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb84815be40 0x7fb8481f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 -- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb84809af90 msgr2=0x7fb8480c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb84809af90 0x7fb8480c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 -- 172.21.15.12:0/3584190144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480cb0c0 con 0x7fb848099ca0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb847fff640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb83800c7b0 tx=0x7fb83800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838018070 con 0x7fb848099ca0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8380072d0 con 0x7fb848099ca0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8480b4650 con 0x7fb848099ca0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380047e0 con 0x7fb848099ca0 2024-03-08T08:30:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.635+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8480cad00 con 0x7fb848099ca0 2024-03-08T08:30:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.636+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8480749c0 con 0x7fb848099ca0 2024-03-08T08:30:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.637+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb83801b070 con 0x7fb848099ca0 2024-03-08T08:30:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.637+0000 7fb844ff9640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 0x7fb83406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.637+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8380894d0 con 0x7fb848099ca0 2024-03-08T08:30:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.637+0000 7fb8477fe640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 0x7fb83406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.637+0000 7fb8477fe640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 0x7fb83406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb83c006fd0 tx=0x7fb83c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.641+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb8480cd890 con 0x7fb848099ca0 2024-03-08T08:30:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.859+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8480ce0f0 con 0x7fb848099ca0 2024-03-08T08:30:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.859+0000 7fb844ff9640 1 -- 172.21.15.12:0/3584190144 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7fb838087e20 con 0x7fb848099ca0 2024-03-08T08:30:17.861 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:30:17.861 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:30:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 msgr2=0x7fb83406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 0x7fb83406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb83c006fd0 tx=0x7fb83c008040 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 msgr2=0x7fb848152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb83800c7b0 tx=0x7fb83800cc70 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 shutdown_connections 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb83406b500 0x7fb83406d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb84815be40 0x7fb8481f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb84809af90 0x7fb8480c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 --2- 172.21.15.12:0/3584190144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb848099ca0 0x7fb848152b10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 >> 172.21.15.12:0/3584190144 conn(0x7fb8480d1900 msgr2=0x7fb8480b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 shutdown_connections 2024-03-08T08:30:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:17.861+0000 7fb84e2ae640 1 -- 172.21.15.12:0/3584190144 wait complete. 2024-03-08T08:30:18.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-08T08:30:12.121225+0000', 'modified': '2024-03-08T08:30:14.788469+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5277}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5277': {'gid': 5277, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6837/2820296658', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6836', 'nonce': 2820296658}, {'type': 'v1', 'addr': '172.21.15.12:6837', 'nonce': 2820296658}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-08T08:30:18.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-08T08:30:18.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-08T08:30:18.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.382+0000 7fd4b4adf640 1 Processor -- start 2024-03-08T08:30:18.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.382+0000 7fd4b4adf640 1 -- start start 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.382+0000 7fd4b4adf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd4b00d1110 0x7fd4b00b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4b4adf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd4b00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4b4adf640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b015e1a0 con 0x7fd4b015ddc0 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4b4adf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b00b9030 con 0x7fd4b00d1110 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4b4adf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00d1d20 con 0x7fd4b00d31c0 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4ae59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd4b015ddc0 0x7fd4b00d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48188/0 (socket says 172.21.15.12:48188) 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4ae59c640 1 -- 172.21.15.12:0/3892769864 learned_addr learned my addr 172.21.15.12:0/3892769864 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714352217 0 0) 0x7fd4b015e1a0 con 0x7fd4b015ddc0 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd49c003520 con 0x7fd4b015ddc0 2024-03-08T08:30:18.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/3892769864 >> v2:172.21.15.156:3300/0 conn(0x7fd4b00d1110 0x7fd4b00b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd4b00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460395205 0 0) 0x7fd49c003520 con 0x7fd4b015ddc0 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 msgr2=0x7fd4b00b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd4b00b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 >> v2:172.21.15.156:3300/0 conn(0x7fd4b00d1110 msgr2=0x7fd4b00b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 --2- 172.21.15.12:0/3892769864 >> v2:172.21.15.156:3300/0 conn(0x7fd4b00d1110 0x7fd4b00b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b00b8790 con 0x7fd4b015ddc0 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fd4a0001670 con 0x7fd4b015ddc0 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/3892769864 >> v2:172.21.15.156:3300/0 conn(0x7fd4b00d1110 0x7fd4b00b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 >> v1:172.21.15.12:6789/0 conn(0x7fd4b015ddc0 legacy=0x7fd4b00d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd49c0039b0 0x7fd49c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd49c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd49c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00d1d20 con 0x7fd49c004d40 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c00ba70 con 0x7fd49c0039b0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c00bbf0 con 0x7fd4b00d31c0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.383+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd49c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd49c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd49c0039b0 0x7fd49c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4aed9d640 1 -- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 msgr2=0x7fd49c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00d31c0 0x7fd49c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4aed9d640 1 -- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd49c0039b0 msgr2=0x7fd49c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd49c0039b0 0x7fd49c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.384+0000 7fd4aed9d640 1 -- 172.21.15.12:0/3892769864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c003520 con 0x7fd49c004d40 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd49c0051c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007390 tx=0x7fd49c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd7387f182004 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8039070 con 0x7fd49c004d40 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a802ccb0 con 0x7fd49c004d40 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4add9b640 1 -- 172.21.15.12:0/3892769864 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8009be0 con 0x7fd49c004d40 2024-03-08T08:30:18.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 -- 172.21.15.12:0/3892769864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 msgr2=0x7fd49c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/3892769864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd49c0051c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007390 tx=0x7fd49c005830 comp rx=0 tx=0).stop 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4aed9d640 1 -- 172.21.15.12:0/3892769864 reap_dead start 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 -- 172.21.15.12:0/3892769864 shutdown_connections 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 -- 172.21.15.12:0/3892769864 >> 172.21.15.12:0/3892769864 conn(0x7fd4b0072720 msgr2=0x7fd4b0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 -- 172.21.15.12:0/3892769864 shutdown_connections 2024-03-08T08:30:18.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.385+0000 7fd4b4adf640 1 -- 172.21.15.12:0/3892769864 wait complete. 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 Processor -- start 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 -- start start 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 0x7fd4b00ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 0x7fd4b00c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00b9980 con 0x7fd49c004d40 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00b8d50 con 0x7fd4b00d2a10 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4b4adf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00b7120 con 0x7fd4b00c98a0 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50172/0 (socket says 172.21.15.12:50172) 2024-03-08T08:30:18.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.387+0000 7fd4ae59c640 1 -- 172.21.15.12:0/540246356 learned_addr learned my addr 172.21.15.12:0/540246356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 0x7fd4b00ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 0x7fd4b00c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 -- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 msgr2=0x7fd4b00c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 0x7fd4b00c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 -- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 msgr2=0x7fd4b00ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 0x7fd4b00ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 -- 172.21.15.12:0/540246356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b00ca260 con 0x7fd49c004d40 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 0x7fd4b00ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 0x7fd4b00c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0007310 tx=0x7fd4a00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4a6ffd640 1 -- 172.21.15.12:0/540246356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000f9c0 con 0x7fd49c004d40 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4a6ffd640 1 -- 172.21.15.12:0/540246356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0018070 con 0x7fd49c004d40 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4a6ffd640 1 -- 172.21.15.12:0/540246356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000fd80 con 0x7fd49c004d40 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b015e430 con 0x7fd49c004d40 2024-03-08T08:30:18.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.388+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b015e9f0 con 0x7fd49c004d40 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.390+0000 7fd4a6ffd640 1 -- 172.21.15.12:0/540246356 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4a001d460 con 0x7fd49c004d40 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.390+0000 7fd4a6ffd640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 0x7fd49006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.390+0000 7fd4a6ffd640 1 -- 172.21.15.12:0/540246356 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4a008a320 con 0x7fd49c004d40 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.390+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 0x7fd49006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.391+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 0x7fd49006d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00cd9a0 tx=0x7fd498008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.391+0000 7fd4b7796740 1 Processor -- start 2024-03-08T08:30:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.391+0000 7fd4b7796740 1 -- start start 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c4158c0 0x557f6c417360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c441a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f6c37b6b0 con 0x557f6c411870 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f6c324a60 con 0x557f6c441600 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f6c377750 con 0x557f6c4158c0 2024-03-08T08:30:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50182/0 (socket says 172.21.15.12:50182) 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4ae59c640 1 -- 172.21.15.12:0/3579955259 learned_addr learned my addr 172.21.15.12:0/3579955259 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c4158c0 0x557f6c417360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c441a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4ae59c640 1 -- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c4158c0 msgr2=0x557f6c417360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.392+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c4158c0 0x557f6c417360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4ae59c640 1 -- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 msgr2=0x557f6c441a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c441a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4ae59c640 1 -- 172.21.15.12:0/3579955259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f6c342f30 con 0x557f6c411870 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c441a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0060c90 tx=0x7fd4a0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c73c8e5f97e8e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd48ffff640 1 -- 172.21.15.12:0/3579955259 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a005cc00 con 0x557f6c411870 2024-03-08T08:30:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd48ffff640 1 -- 172.21.15.12:0/3579955259 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a00611c0 con 0x557f6c411870 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd48ffff640 1 -- 172.21.15.12:0/3579955259 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0056db0 con 0x557f6c411870 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4b7796740 1 -- 172.21.15.12:0/3579955259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 msgr2=0x557f6c411c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.393+0000 7fd4b7796740 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0060c90 tx=0x7fd4a0000fe0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 -- 172.21.15.12:0/3579955259 shutdown_connections 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c441a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c4158c0 0x557f6c417360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 --2- 172.21.15.12:0/3579955259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c411870 0x557f6c411c50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 -- 172.21.15.12:0/3579955259 >> 172.21.15.12:0/3579955259 conn(0x557f6c4138f0 msgr2=0x557f6c4151b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 -- 172.21.15.12:0/3579955259 shutdown_connections 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 -- 172.21.15.12:0/3579955259 wait complete. 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 -- start start 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.394+0000 7fd4b7796740 1 Processor -- start 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c441600 0x557f6c434110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c435330 0x557f6c437710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f6c415d50 con 0x557f6c434d10 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f6c337ea0 con 0x557f6c435330 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f6c416030 con 0x557f6c441600 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4b7796740 10 client.0 _ll_register_callbacks cb 0x557f6c337870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4aed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4aed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50196/0 (socket says 172.21.15.12:50196) 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4aed9d640 1 -- 172.21.15.12:0/1259324772 learned_addr learned my addr 172.21.15.12:0/1259324772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4aed9d640 1 -- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c441600 msgr2=0x557f6c434110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.395+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c441600 0x557f6c434110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4aed9d640 1 -- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c435330 msgr2=0x557f6c437710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c435330 0x557f6c437710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4aed9d640 1 -- 172.21.15.12:0/1259324772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f6c338270 con 0x557f6c434d10 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80074b0 tx=0x7fd4a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bab129c69facaf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8044060 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4a8034850 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4a00611c0 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4a804a050 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 10 client.5448 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x557f6c0902d0 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a00611c0 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd4a804d020 con 0x557f6c434d10 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 10 client.5448 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 10 client.5448 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 20 client.5448 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c444350 0x557f6c47bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 4 client.5448 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f6c388bf0 con 0x557f6c444350 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.396+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c444350 0x557f6c47bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd48d7fa640 10 client.5448 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c444350 0x557f6c47bc70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eda519922dd2e01a server_cookie=ade610127ebcdc6a in_seq=0 out_seq=0 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd48d7fa640 1 -- 172.21.15.12:0/1259324772 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557f6c388bf0 con 0x557f6c444350 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd48d7fa640 10 client.5448 handle_command_reply: tid=0 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd4b7796740 1 client.5448 shutdown 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 msgr2=0x557f6c437c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.397+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80074b0 tx=0x7fd4a8004a40 comp rx=0 tx=0).stop 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 shutdown_connections 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c444350 0x557f6c47bc70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c437c50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c435330 0x557f6c437710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 --2- 172.21.15.12:0/1259324772 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c441600 0x557f6c434110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd48d7fa640 0 client.5448 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 >> 172.21.15.12:0/1259324772 conn(0x557f6c411870 msgr2=0x557f6c43f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 shutdown_connections 2024-03-08T08:30:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 1 -- 172.21.15.12:0/1259324772 wait complete. 2024-03-08T08:30:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.398+0000 7fd4b7796740 20 client.5448 trim_cache size 0 max 16384 2024-03-08T08:30:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.413+0000 7fd4b7796740 1 Processor -- start 2024-03-08T08:30:18.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 -- start start 2024-03-08T08:30:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c45fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c460010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0200330 con 0x557f6c434d10 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b01ff560 con 0x557f6c441600 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b015ff80 con 0x557f6c435330 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50206/0 (socket says 172.21.15.12:50206) 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.414+0000 7fd4ae59c640 1 -- 172.21.15.12:0/681225089 learned_addr learned my addr 172.21.15.12:0/681225089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 -- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 msgr2=0x557f6c45fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c45fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 -- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 msgr2=0x557f6c460010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c460010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 -- 172.21.15.12:0/681225089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f6c15a6b0 con 0x557f6c434d10 2024-03-08T08:30:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0058a40 tx=0x7fd4a0077000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605cd8e05dd2fbd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd48f7fe640 1 -- 172.21.15.12:0/681225089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a00611c0 con 0x557f6c434d10 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd48f7fe640 1 -- 172.21.15.12:0/681225089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a008a0c0 con 0x557f6c434d10 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd48f7fe640 1 -- 172.21.15.12:0/681225089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0055120 con 0x557f6c434d10 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 -- 172.21.15.12:0/681225089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 msgr2=0x557f6c434110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0058a40 tx=0x7fd4a0077000 comp rx=0 tx=0).stop 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 -- 172.21.15.12:0/681225089 shutdown_connections 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c441600 0x557f6c460010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c45fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 --2- 172.21.15.12:0/681225089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c434d10 0x557f6c434110 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 -- 172.21.15.12:0/681225089 >> 172.21.15.12:0/681225089 conn(0x557f6c411870 msgr2=0x557f6c410c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 -- 172.21.15.12:0/681225089 shutdown_connections 2024-03-08T08:30:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.415+0000 7fd4b7796740 1 -- 172.21.15.12:0/681225089 wait complete. 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.416+0000 7fd4b7796740 1 -- start start 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.416+0000 7fd4b7796740 1 Processor -- start 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 0x557f6c47d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c4ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f6c434b30 con 0x557f6c441600 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0160580 con 0x557f6c434d10 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f6c444d40 con 0x557f6c435330 2024-03-08T08:30:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4b7796740 10 client.0 _ll_register_callbacks cb 0x7fd4a8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50212/0 (socket says 172.21.15.12:50212) 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 -- 172.21.15.12:0/655265916 learned_addr learned my addr 172.21.15.12:0/655265916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 0x557f6c47d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4a7fff640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c4ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 -- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 msgr2=0x557f6c4ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c4ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 -- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 msgr2=0x557f6c47d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 0x557f6c47d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 -- 172.21.15.12:0/655265916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f6c47d880 con 0x557f6c441600 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 0x557f6c47d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd4aed9d640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80082b0 tx=0x7fd4a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c9fe40217ee475 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.417+0000 7fd48d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8054060 con 0x557f6c441600 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4a8034860 con 0x557f6c441600 2024-03-08T08:30:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4a008a0c0 con 0x557f6c441600 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4a805a050 con 0x557f6c441600 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 10 client.5454 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x557f6c4406b0 con 0x557f6c441600 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a008a0c0 con 0x557f6c441600 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd4a805d020 con 0x557f6c441600 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 10 client.5454 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 10 client.5454 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 20 client.5454 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c438040 0x557f6c4419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 4 client.5454 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557f6c414050 con 0x557f6c438040 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c438040 0x557f6c4419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd48d7fa640 10 client.5454 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.418+0000 7fd4ae59c640 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c438040 0x557f6c4419e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13256f86b0e465ab server_cookie=2e4e56e1f91b7620 in_seq=0 out_seq=0 2024-03-08T08:30:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd48d7fa640 1 -- 172.21.15.12:0/655265916 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x557f6c414050 con 0x557f6c438040 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd48d7fa640 10 client.5454 handle_command_reply: tid=0 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 client.5454 shutdown 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 msgr2=0x557f6c4ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80082b0 tx=0x7fd4a8004ab0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 shutdown_connections 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x557f6c438040 0x557f6c4419e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f6c441600 0x557f6c4ca9e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x557f6c435330 0x557f6c4ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 --2- 172.21.15.12:0/655265916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f6c434d10 0x557f6c47d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd48d7fa640 0 client.5454 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 >> 172.21.15.12:0/655265916 conn(0x557f6c411870 msgr2=0x557f6c43f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 shutdown_connections 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 1 -- 172.21.15.12:0/655265916 wait complete. 2024-03-08T08:30:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b7796740 20 client.5454 trim_cache size 0 max 16384 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c", 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5277, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 5.665858976, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 15, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 35, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 35, 2024-03-08T08:30:18.427 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 13.575537224 2024-03-08T08:30:18.428 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 msgr2=0x7fd49006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 0x7fd49006d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00cd9a0 tx=0x7fd498008040 comp rx=0 tx=0).stop 2024-03-08T08:30:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 msgr2=0x7fd4b0154200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0007310 tx=0x7fd4a00077d0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 shutdown_connections 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd49006b4d0 0x7fd49006d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.419+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4b00c98a0 0x7fd4b00c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.420+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b00d2a10 0x7fd4b00ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.420+0000 7fd4b4adf640 1 --2- 172.21.15.12:0/540246356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd49c004d40 0x7fd4b0154200 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.420+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 >> 172.21.15.12:0/540246356 conn(0x7fd4b0072720 msgr2=0x7fd4b00d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.420+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 shutdown_connections 2024-03-08T08:30:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.420+0000 7fd4b4adf640 1 -- 172.21.15.12:0/540246356 wait complete. 2024-03-08T08:30:18.432 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:30:18.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.531+0000 7f8856e07640 1 Processor -- start 2024-03-08T08:30:18.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.531+0000 7f8856e07640 1 -- start start 2024-03-08T08:30:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8856e07640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f8850160000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8856e07640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500b2df0 con 0x7f885015f200 2024-03-08T08:30:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8856e07640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500d01a0 con 0x7f88500b14d0 2024-03-08T08:30:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8856e07640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8850099b90 con 0x7f88500b2830 2024-03-08T08:30:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8854ba3640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f885015f200 0x7f885015f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52382/0 (socket says 172.21.15.12:52382) 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f8854ba3640 1 -- 172.21.15.12:0/3307161411 learned_addr learned my addr 172.21.15.12:0/3307161411 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.532+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960651398 0 0) 0x7f88500b2df0 con 0x7f885015f200 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884ffff640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f8850160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8838003520 con 0x7f885015f200 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645902331 0 0) 0x7f8838003520 con 0x7f885015f200 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 msgr2=0x7f8850160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f8850160000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 >> v1:172.21.15.156:6789/0 conn(0x7f88500b14d0 legacy=0x7f88500b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500d0480 con 0x7f885015f200 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f88480019f0 con 0x7f885015f200 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 >> v1:172.21.15.12:6789/0 conn(0x7f885015f200 legacy=0x7f885015f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88380039e0 0x7f8838003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8838004390 0x7f8838004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f88380050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883800b9c0 con 0x7f88380039e0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883800bb40 con 0x7f8838004390 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883800bd40 con 0x7f88500b2830 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88380039e0 0x7f8838003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884ffff640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8838004390 0x7f8838004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f88553a4640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f88380050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 -- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 msgr2=0x7f88380050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f88380050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 -- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8838004390 msgr2=0x7f8838004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8838004390 0x7f8838004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f8854ba3640 1 -- 172.21.15.12:0/3307161411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838003520 con 0x7f88380039e0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f884ffff640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8838004390 0x7f8838004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.533+0000 7f88553a4640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88500b2830 0x7f88380050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.534+0000 7f8854ba3640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88380039e0 0x7f8838003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f884800a920 tx=0x7f8838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4130ecdee15b5bb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.534+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884800a4c0 con 0x7f88380039e0 2024-03-08T08:30:18.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.534+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8848034650 con 0x7f88380039e0 2024-03-08T08:30:18.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.534+0000 7f884f7fe640 1 -- 172.21.15.12:0/3307161411 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848034a40 con 0x7f88380039e0 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 -- 172.21.15.12:0/3307161411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88380039e0 msgr2=0x7f8838003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 --2- 172.21.15.12:0/3307161411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88380039e0 0x7f8838003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f884800a920 tx=0x7f8838005830 comp rx=0 tx=0).stop 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f88553a4640 1 -- 172.21.15.12:0/3307161411 reap_dead start 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 -- 172.21.15.12:0/3307161411 shutdown_connections 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 -- 172.21.15.12:0/3307161411 >> 172.21.15.12:0/3307161411 conn(0x7f88500b3460 msgr2=0x7f88500b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 -- 172.21.15.12:0/3307161411 shutdown_connections 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.535+0000 7f8856e07640 1 -- 172.21.15.12:0/3307161411 wait complete. 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.536+0000 7f8856e07640 1 Processor -- start 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.536+0000 7f8856e07640 1 -- start start 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 0x7f8850150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8850158130 0x7f88500b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500d07f0 con 0x7f88500c72c0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500cff50 con 0x7f88380039e0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88500cd720 con 0x7f8850158130 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41656/0 (socket says 172.21.15.12:41656) 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 -- 172.21.15.12:0/377003378 learned_addr learned my addr 172.21.15.12:0/377003378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f884ffff640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8850158130 0x7f88500b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8854ba3640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 0x7f8850150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 -- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8850158130 msgr2=0x7f88500b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8850158130 0x7f88500b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 -- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 msgr2=0x7f8850150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 0x7f8850150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 -- 172.21.15.12:0/377003378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500c6570 con 0x7f88500c72c0 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8854ba3640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 0x7f8850150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f88553a4640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88500cf450 tx=0x7f884000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840019070 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88400052e0 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88500c67e0 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.537+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840003810 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.538+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88500b4d00 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.539+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88500749c0 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.539+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88400039b0 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.539+0000 7f884d7fa640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 0x7f883c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.539+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88400895a0 con 0x7f88500c72c0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.539+0000 7f8854ba3640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 0x7f883c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.540+0000 7f8854ba3640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 0x7f883c06d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f884800a920 tx=0x7f884802f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:18.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.544+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8840057dd0 con 0x7f88500c72c0 2024-03-08T08:30:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.760+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f88500cbca0 con 0x7f88500c72c0 2024-03-08T08:30:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.760+0000 7f884d7fa640 1 -- 172.21.15.12:0/377003378 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f884005be70 con 0x7f88500c72c0 2024-03-08T08:30:18.763 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:30:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 msgr2=0x7f883c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 0x7f883c06d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f884800a920 tx=0x7f884802f000 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 msgr2=0x7f88500c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88500cf450 tx=0x7f884000e960 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 shutdown_connections 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f883c06b500 0x7f883c06d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88500c72c0 0x7f88500c7770 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8850158130 0x7f88500b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 --2- 172.21.15.12:0/377003378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88380039e0 0x7f8850150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 >> 172.21.15.12:0/377003378 conn(0x7f88500b3460 msgr2=0x7f88500b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 shutdown_connections 2024-03-08T08:30:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:18.762+0000 7f8856e07640 1 -- 172.21.15.12:0/377003378 wait complete. 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:30:18.775 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-08T08:30:18.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:18.776 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-08T08:30:18.790 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:30:18.790 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:18.791 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42554sec preferred_lft 42554sec 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 76:bf:65:0f:db:fc brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-08T08:30:18.792 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::a4b2:1eff:fe40:f0ec/64 scope link 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 76:bf:65:0f:db:fc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::74bf:65ff:fe0f:dbfc/64 scope link 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether aa:74:ca:4a:90:e0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::a874:caff:fe4a:90e0/64 scope link 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:18.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:18.793 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:30:18.856 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-08T08:30:18.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:18.858 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-08T08:30:18.913 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-08T08:30:18.913 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:30:18.913 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-08T08:30:18.913 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-08T08:30:18.914 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:30:18.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-08T08:30:18.914 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-08T08:30:19.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.009 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-08T08:30:19.023 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-08T08:30:19.024 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-08T08:30:19.024 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-08T08:30:19.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-08T08:30:19.090 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-08T08:30:19.090 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:30:19.090 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:19.144 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-08T08:30:19.144 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:30:19.144 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:19.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:30:19.265 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:30:19.320 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:30:19.321 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:30:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:30:19.323 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:30:19.323 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:30:19.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.323 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:30:19.375 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-08T08:30:19.375 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-08T08:30:19.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-08T08:30:19.445 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:30:19.501 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-08T08:30:19.501 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-08T08:30:19.501 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-08T08:30:19.501 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-08T08:30:19.501 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-08T08:30:19.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:30:19.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:30:19.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.504 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-08T08:30:19.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-03-08T08:30:19.503+0000 7f17be04d480 -1 init, newargv = 0x5602ab4fd460 newargc=15 2024-03-08T08:30:19.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[51735]: starting ceph client 2024-03-08T08:30:19.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[51735]: starting fuse 2024-03-08T08:30:19.520 INFO:teuthology.orchestra.run.smithi012.stdout:41 2024-03-08T08:30:19.520 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-03-08T08:30:19.520 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:19.522 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-03-08T08:30:19.522 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-08T08:30:19.641 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.51735.asok 2024-03-08T08:30:19.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:30:19.651 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51735.asok 2024-03-08T08:30:19.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51735.asok status 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "51735", 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5466, 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5466 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1928274948 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1928274948 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.5466 192.168.0.1:0/1928274948", 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/1928274948", 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 15, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 35, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-03-08T08:30:19.793 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:19.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.800 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-03-08T08:30:19.800 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:19.814 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-03-08T08:30:19.814 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:19.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:19.814 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-08T08:30:19.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-08T08:30:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.020+0000 7f61c490c640 1 Processor -- start 2024-03-08T08:30:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.020+0000 7f61c490c640 1 -- start start 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61c490c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f61c009a580 0x7f61c00cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61c490c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c009a960 con 0x7f61c006ce00 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61c490c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00d0730 con 0x7f61c00d00b0 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61c490c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c00d0a10 con 0x7f61c009a580 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f61c006ce00 0x7f61c00cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52396/0 (socket says 172.21.15.12:52396) 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bed9d640 1 -- 172.21.15.12:0/4058695146 learned_addr learned my addr 172.21.15.12:0/4058695146 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61be59c640 1 --2- 172.21.15.12:0/4058695146 >> v2:172.21.15.156:3300/0 conn(0x7f61c009a580 0x7f61c00cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867980704 0 0) 0x7f61c009a960 con 0x7f61c006ce00 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003520 con 0x7f61c006ce00 2024-03-08T08:30:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 788668988 0 0) 0x7f61ac003520 con 0x7f61c006ce00 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 >> v2:172.21.15.156:3300/0 conn(0x7f61c009a580 msgr2=0x7f61c00cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 --2- 172.21.15.12:0/4058695146 >> v2:172.21.15.156:3300/0 conn(0x7f61c009a580 0x7f61c00cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 >> v1:172.21.15.156:6789/0 conn(0x7f61c00d00b0 legacy=0x7f61c006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00d0cf0 con 0x7f61c006ce00 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f61b80019f0 con 0x7f61c006ce00 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 >> v1:172.21.15.12:6789/0 conn(0x7f61c006ce00 legacy=0x7f61c00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61ac0039b0 0x7f61ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ac004800 0x7f61ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c00d0a10 con 0x7f61ac0051c0 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00c060 con 0x7f61ac004800 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.021+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00c260 con 0x7f61ac0039b0 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61be59c640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61ac0039b0 0x7f61ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61b7fff640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ac004800 0x7f61ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 -- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61ac0039b0 msgr2=0x7f61ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61ac0039b0 0x7f61ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 -- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ac004800 msgr2=0x7f61ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ac004800 0x7f61ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61bed9d640 1 -- 172.21.15.12:0/4058695146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac003520 con 0x7f61ac0051c0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.022+0000 7f61b7fff640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ac004800 0x7f61ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61bed9d640 1 -- 172.21.15.12:0/4058695146 reap_dead start 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61bed9d640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61ac00b9c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f61c00b33f0 tx=0x7f61ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbc2c2ed904a7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b800a4a0 con 0x7f61ac0051c0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b8034650 con 0x7f61ac0051c0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61bdd9b640 1 -- 172.21.15.12:0/4058695146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8034a40 con 0x7f61ac0051c0 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 -- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 msgr2=0x7f61ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61ac00b9c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f61c00b33f0 tx=0x7f61ac005830 comp rx=0 tx=0).stop 2024-03-08T08:30:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 -- 172.21.15.12:0/4058695146 shutdown_connections 2024-03-08T08:30:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 --2- 172.21.15.12:0/4058695146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61ac00b9c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 -- 172.21.15.12:0/4058695146 >> 172.21.15.12:0/4058695146 conn(0x7f61c00b3420 msgr2=0x7f61c00b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 -- 172.21.15.12:0/4058695146 shutdown_connections 2024-03-08T08:30:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.023+0000 7f61c490c640 1 -- 172.21.15.12:0/4058695146 wait complete. 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 Processor -- start 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 -- start start 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c0156750 0x7f61c00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c015c9a0 0x7f61c015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c006c120 con 0x7f61ac0051c0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c00d12b0 con 0x7f61c015c9a0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61c490c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c006c2a0 con 0x7f61c0156750 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61be59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61be59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41684/0 (socket says 172.21.15.12:41684) 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.025+0000 7f61be59c640 1 -- 172.21.15.12:0/366066333 learned_addr learned my addr 172.21.15.12:0/366066333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61bed9d640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c015c9a0 0x7f61c015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61b7fff640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c0156750 0x7f61c00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 -- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c0156750 msgr2=0x7f61c00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c0156750 0x7f61c00b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 -- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c015c9a0 msgr2=0x7f61c015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c015c9a0 0x7f61c015ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 -- 172.21.15.12:0/366066333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00b3230 con 0x7f61ac0051c0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61be59c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f61b0011910 tx=0x7f61b0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b000fa90 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b000fc30 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c01f6f20 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0014440 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.026+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c01f74e0 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.027+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c006cc20 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.027+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61b0012040 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.028+0000 7f61b6ffd640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 0x7f61a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.028+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61b008a730 con 0x7f61ac0051c0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.028+0000 7f61b7fff640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 0x7f61a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.028+0000 7f61b7fff640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 0x7f61a006d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f61a8000c00 tx=0x7f61a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.032+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f61c006cc20 con 0x7f61ac0051c0 2024-03-08T08:30:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.255+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f61c00749c0 con 0x7f61ac0051c0 2024-03-08T08:30:20.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.256+0000 7f61b6ffd640 1 -- 172.21.15.12:0/366066333 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f61b0019030 con 0x7f61ac0051c0 2024-03-08T08:30:20.258 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.257+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 msgr2=0x7f61a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.257+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 0x7f61a006d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f61a8000c00 tx=0x7f61a8009040 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.257+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 msgr2=0x7f61c015ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.257+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f61b0011910 tx=0x7f61b0011dd0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.257+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 shutdown_connections 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f61a006b500 0x7f61a006d9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c015c9a0 0x7f61c015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c0156750 0x7f61c00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 --2- 172.21.15.12:0/366066333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61ac0051c0 0x7f61c015ab80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 >> 172.21.15.12:0/366066333 conn(0x7f61c00b3420 msgr2=0x7f61c00b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 shutdown_connections 2024-03-08T08:30:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:20.258+0000 7f61c490c640 1 -- 172.21.15.12:0/366066333 wait complete. 2024-03-08T08:30:20.270 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T08:30:20.270 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T08:30:20.270 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T08:30:20.271 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-08T08:30:20.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.271 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:20.291 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42548sec preferred_lft 42548sec 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether a2:50:32:30:3f:e5 brd ff:ff:ff:ff:ff:ff 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::d091:12ff:fe72:2ccf/64 scope link 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether a2:50:32:30:3f:e5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:30:20.292 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::a050:32ff:fe30:3fe5/64 scope link 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run.smithi156.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether ae:67:7b:f2:d6:5b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ac67:7bff:fef2:d65b/64 scope link 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-08T08:30:20.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.293 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T08:30:20.363 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-08T08:30:20.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.364 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-08T08:30:20.423 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-08T08:30:20.423 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T08:30:20.424 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-08T08:30:20.424 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-08T08:30:20.424 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T08:30:20.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-08T08:30:20.424 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-08T08:30:20.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.524 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-08T08:30:20.539 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-08T08:30:20.539 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-08T08:30:20.539 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-08T08:30:20.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.539 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2024-03-08T08:30:20.606 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-08T08:30:20.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:30:20.607 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:20.662 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-08T08:30:20.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T08:30:20.662 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:20.717 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:30:20.786 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:30:20.841 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:30:20.842 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:30:20.843 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:30:20.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:20.844 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:30:20.896 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-08T08:30:20.896 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-08T08:30:20.938 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-08T08:30:20.964 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-08T08:30:21.020 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T08:30:21.021 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:30:21.022 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:30:21.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:21.023 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-08T08:30:21.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2024-03-08T08:30:21.023+0000 7f782e80b480 -1 init, newargv = 0x5641b166c040 newargc=15 2024-03-08T08:30:21.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[43673]: starting ceph client 2024-03-08T08:30:21.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[43673]: starting fuse 2024-03-08T08:30:21.039 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-08T08:30:21.039 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-08T08:30:21.039 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:30:21.039 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-08T08:30:21.039 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:30:21.040 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2024-03-08T08:30:21.041 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:21.161 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.43673.asok 2024-03-08T08:30:21.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:30:21.171 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43673.asok 2024-03-08T08:30:21.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:21.171 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43673.asok status 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "1fb5d10608dccee73edd6ef8a2c7b804a097c324", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "43673", 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:21.308 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4796, 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4796 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2054950743 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:30:21.309 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2054950743 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4796 192.168.0.1:0/2054950743", 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/2054950743", 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 15, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 35, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-08T08:30:21.310 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-08T08:30:21.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:21.316 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-08T08:30:21.316 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:21.331 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-08T08:30:21.331 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:21.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:21.331 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-08T08:30:31.399 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T08:30:31.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 counter dump 2024-03-08T08:30:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.513+0000 7f8d18ed4640 1 Processor -- start 2024-03-08T08:30:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.513+0000 7f8d18ed4640 1 -- start start 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d18ed4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d14168cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d18ed4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d140b6be0 con 0x7f8d1415dec0 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d18ed4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1415e2a0 con 0x7f8d140d0250 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d18ed4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140d2140 con 0x7f8d140d35e0 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d1259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8d1415dec0 0x7f8d14161990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46604/0 (socket says 172.21.15.12:46604) 2024-03-08T08:30:31.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d1259c640 1 -- 172.21.15.12:0/3689492627 learned_addr learned my addr 172.21.15.12:0/3689492627 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:31.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.514+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076403421 0 0) 0x7f8d0c001110 con 0x7f8d1415dec0 2024-03-08T08:30:31.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d00003520 con 0x7f8d1415dec0 2024-03-08T08:30:31.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d14168cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677248483 0 0) 0x7f8d00003520 con 0x7f8d1415dec0 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 msgr2=0x7f8d14168cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d14168cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 >> v1:172.21.15.156:6789/0 conn(0x7f8d140d0250 legacy=0x7f8d140d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d141620a0 con 0x7f8d1415dec0 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f8d0c001860 con 0x7f8d1415dec0 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 >> v1:172.21.15.12:6789/0 conn(0x7f8d1415dec0 legacy=0x7f8d14161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d00003f60 0x7f8d00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140d2140 con 0x7f8d00004d40 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0000baa0 con 0x7f8d00003f60 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0000bca0 con 0x7f8d140d35e0 2024-03-08T08:30:31.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d00003f60 0x7f8d00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 -- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 msgr2=0x7f8d000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140d35e0 0x7f8d000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 -- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d00003f60 msgr2=0x7f8d00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d00003f60 0x7f8d00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.515+0000 7f8d12d9d640 1 -- 172.21.15.12:0/3689492627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00003520 con 0x7f8d00004d40 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.516+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d000051c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009fc0 tx=0x7f8d00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22da18c12db691d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.516+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04039070 con 0x7f8d00004d40 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.516+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0402cc20 con 0x7f8d00004d40 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.516+0000 7f8d1159a640 1 -- 172.21.15.12:0/3689492627 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04009c10 con 0x7f8d00004d40 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3689492627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 msgr2=0x7f8d000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3689492627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d000051c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009fc0 tx=0x7f8d00005830 comp rx=0 tx=0).stop 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d12d9d640 1 -- 172.21.15.12:0/3689492627 reap_dead start 2024-03-08T08:30:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3689492627 shutdown_connections 2024-03-08T08:30:31.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3689492627 >> 172.21.15.12:0/3689492627 conn(0x7f8d1406c240 msgr2=0x7f8d1406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3689492627 shutdown_connections 2024-03-08T08:30:31.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.517+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3689492627 wait complete. 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.518+0000 7f8d18ed4640 1 Processor -- start 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.518+0000 7f8d18ed4640 1 -- start start 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d1415f590 0x7f8d141f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 0x7f8d140b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140d1dd0 con 0x7f8d00004d40 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14162630 con 0x7f8d1415f590 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d18ed4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140b94f0 con 0x7f8d140b56c0 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d1259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d1259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47848/0 (socket says 172.21.15.12:47848) 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d1259c640 1 -- 172.21.15.12:0/3922281644 learned_addr learned my addr 172.21.15.12:0/3922281644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 0x7f8d140b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d1415f590 0x7f8d141f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d1259c640 1 -- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 msgr2=0x7f8d140b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.519+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 0x7f8d140b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d1259c640 1 -- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d1415f590 msgr2=0x7f8d141f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d1415f590 0x7f8d141f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d1259c640 1 -- 172.21.15.12:0/3922281644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d141f9300 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 0x7f8d140b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0019d0 tx=0x7f8d0c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d0affd640 1 -- 172.21.15.12:0/3922281644 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c007740 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d0affd640 1 -- 172.21.15.12:0/3922281644 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c018070 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1415fb50 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d0affd640 1 -- 172.21.15.12:0/3922281644 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c00fc00 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.520+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d14160110 con 0x7f8d00004d40 2024-03-08T08:30:31.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.522+0000 7f8d0affd640 1 -- 172.21.15.12:0/3922281644 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d0c01d460 con 0x7f8d00004d40 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.522+0000 7f8d0affd640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.522+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.522+0000 7f8d0affd640 1 -- 172.21.15.12:0/3922281644 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d0c08a280 con 0x7f8d00004d40 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.523+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 0x7f8cf406d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d140b8b30 tx=0x7f8cfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.523+0000 7f8d1bb93740 1 Processor -- start 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.523+0000 7f8d1bb93740 1 -- start start 2024-03-08T08:30:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f52638a0 0x5559f5263c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5293ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559f51cde30 con 0x5559f52678f0 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559f5176d20 con 0x5559f5293630 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x5559f51c9ed0 con 0x5559f52638a0 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47852/0 (socket says 172.21.15.12:47852) 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 -- 172.21.15.12:0/1429515066 learned_addr learned my addr 172.21.15.12:0/1429515066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 -- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f52638a0 msgr2=0x5559f5263c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f52638a0 0x5559f5263c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 -- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 msgr2=0x5559f5293ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5293ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.524+0000 7f8d11d9b640 1 -- 172.21.15.12:0/1429515066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559f4ee23f0 con 0x5559f52678f0 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0018b0 tx=0x7f8cfc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e7034b2979612e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8cf3fff640 1 -- 172.21.15.12:0/1429515066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc028070 con 0x5559f52678f0 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8cf3fff640 1 -- 172.21.15.12:0/1429515066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc015c00 con 0x5559f52678f0 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8cf3fff640 1 -- 172.21.15.12:0/1429515066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc0143d0 con 0x5559f52678f0 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8d1bb93740 1 -- 172.21.15.12:0/1429515066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 msgr2=0x5559f5269390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.525+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0018b0 tx=0x7f8cfc014000 comp rx=0 tx=0).stop 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 -- 172.21.15.12:0/1429515066 shutdown_connections 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5293ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f52678f0 0x5559f5269390 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/1429515066 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f52638a0 0x5559f5263c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 -- 172.21.15.12:0/1429515066 >> 172.21.15.12:0/1429515066 conn(0x5559f5265920 msgr2=0x5559f52671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 -- 172.21.15.12:0/1429515066 shutdown_connections 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 -- 172.21.15.12:0/1429515066 wait complete. 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 -- start start 2024-03-08T08:30:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.526+0000 7f8d1bb93740 1 Processor -- start 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f52863f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5295860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559f5267d80 con 0x5559f528bdc0 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559f516c390 con 0x5559f5293630 2024-03-08T08:30:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x5559f5268060 con 0x5559f5286a60 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1bb93740 10 client.0 _ll_register_callbacks cb 0x5559f5189c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47866/0 (socket says 172.21.15.12:47866) 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d11d9b640 1 -- 172.21.15.12:0/3733071627 learned_addr learned my addr 172.21.15.12:0/3733071627 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.527+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f52863f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5295860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 -- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 msgr2=0x5559f5295860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5295860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 -- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 msgr2=0x5559f52863f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f52863f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 -- 172.21.15.12:0/3733071627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559f4fac7d0 con 0x5559f528bdc0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5295860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5559f5297d20 tx=0x7f8cfc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf6a650d66a556f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc028070 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cfc022da0 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cfc015c00 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cfc03b050 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d1bb93740 10 client.5487 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x5559f5295da0 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc015c00 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.528+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8cfc03e020 con 0x5559f528bdc0 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 10 client.5487 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 10 client.5487 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 20 client.5487 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f52b1850 0x5559f52b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 4 client.5487 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559f51db370 con 0x5559f52b1850 2024-03-08T08:30:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f52b1850 0x5559f52b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8cf17fa640 10 client.5487 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.529+0000 7f8d1259c640 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f52b1850 0x5559f52b1c60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb4a7524dd7b2cb6 server_cookie=9406f65bda16adfa in_seq=0 out_seq=0 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8cf17fa640 1 -- 172.21.15.12:0/3733071627 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5559f51db370 con 0x5559f52b1850 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8cf17fa640 10 client.5487 handle_command_reply: tid=0 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 client.5487 shutdown 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 msgr2=0x5559f5295320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5559f5297d20 tx=0x7f8cfc001120 comp rx=0 tx=0).stop 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 shutdown_connections 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f52b1850 0x5559f52b1c60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5295860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5295320 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/3733071627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f52863f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8cf17fa640 0 client.5487 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 >> 172.21.15.12:0/3733071627 conn(0x5559f52638a0 msgr2=0x5559f5292730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 shutdown_connections 2024-03-08T08:30:31.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 1 -- 172.21.15.12:0/3733071627 wait complete. 2024-03-08T08:30:31.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.530+0000 7f8d1bb93740 20 client.5487 trim_cache size 0 max 16384 2024-03-08T08:30:31.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 Processor -- start 2024-03-08T08:30:31.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 -- start start 2024-03-08T08:30:31.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 0x5559f52afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5293630 0x5559f52b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14204b00 con 0x5559f5286a60 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14203cf0 con 0x5559f528bdc0 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.542+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d141e7fc0 con 0x5559f5293630 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47880/0 (socket says 172.21.15.12:47880) 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 -- 172.21.15.12:0/348734311 learned_addr learned my addr 172.21.15.12:0/348734311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5293630 0x5559f52b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 0x5559f52afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 -- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5293630 msgr2=0x5559f52b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5293630 0x5559f52b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 -- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 msgr2=0x5559f52afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 0x5559f52afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 -- 172.21.15.12:0/348734311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559f52bf090 con 0x5559f5286a60 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 0x5559f52afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:31.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8d1259c640 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0013f0 tx=0x7f8d0c08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869283755fa13466 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8cf37fe640 1 -- 172.21.15.12:0/348734311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c056aa0 con 0x5559f5286a60 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8cf37fe640 1 -- 172.21.15.12:0/348734311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c056d00 con 0x5559f5286a60 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.543+0000 7f8cf37fe640 1 -- 172.21.15.12:0/348734311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c055070 con 0x5559f5286a60 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- 172.21.15.12:0/348734311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 msgr2=0x5559f5269550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0013f0 tx=0x7f8d0c08a040 comp rx=0 tx=0).stop 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- 172.21.15.12:0/348734311 shutdown_connections 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5293630 0x5559f52b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f528bdc0 0x5559f52afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/348734311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f5286a60 0x5559f5269550 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- 172.21.15.12:0/348734311 >> 172.21.15.12:0/348734311 conn(0x5559f52638a0 msgr2=0x5559f52b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- 172.21.15.12:0/348734311 shutdown_connections 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- 172.21.15.12:0/348734311 wait complete. 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 -- start start 2024-03-08T08:30:31.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.544+0000 7f8d1bb93740 1 Processor -- start 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5283250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5269990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559f5265470 con 0x5559f528bdc0 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d141e85c0 con 0x5559f5293630 2024-03-08T08:30:31.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x5559f5287030 con 0x5559f5286a60 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d1bb93740 10 client.0 _ll_register_callbacks cb 0x7f8cfc026070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.545+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47882/0 (socket says 172.21.15.12:47882) 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 -- 172.21.15.12:0/4143653505 learned_addr learned my addr 172.21.15.12:0/4143653505 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d1259c640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5283250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d12d9d640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5269990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 -- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 msgr2=0x5559f5283250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5283250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 -- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 msgr2=0x5559f5269990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5269990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 -- 172.21.15.12:0/4143653505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559f5288f70 con 0x5559f528bdc0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d11d9b640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc003e50 tx=0x7f8cfc00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2dd1dd8ba4596 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:31.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc043020 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cfc022da0 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d0c056d00 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cfc04d050 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d1bb93740 10 client.5493 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x5559f52b2740 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.546+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c056d00 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8cfc050020 con 0x5559f528bdc0 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 10 client.5493 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 10 client.5493 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 20 client.5493 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f5319ad0 0x5559f5269ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 4 client.5493 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "counter dump"}] 2024-03-08T08:30:31.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "counter dump"}) v1 -- 0x5559f5264660 con 0x5559f5319ad0 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1259c640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f5319ad0 0x5559f5269ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8cf17fa640 10 client.5493 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.547+0000 7f8d1259c640 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f5319ad0 0x5559f5269ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b594c85b9fa258 server_cookie=c58d238e54014c2c in_seq=0 out_seq=0 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8cf17fa640 1 -- 172.21.15.12:0/4143653505 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26739 (crc 0 0 0) 0x5559f5264660 con 0x5559f5319ad0 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8cf17fa640 10 client.5493 handle_command_reply: tid=0 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 client.5493 shutdown 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 msgr2=0x5559f5283790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc003e50 tx=0x7f8cfc00c000 comp rx=0 tx=0).stop 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 shutdown_connections 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x5559f5319ad0 0x5559f5269ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559f5293630 0x5559f5269990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559f528bdc0 0x5559f5283790 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 --2- 172.21.15.12:0/4143653505 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x5559f5286a60 0x5559f5283250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8cf17fa640 0 client.5493 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 >> 172.21.15.12:0/4143653505 conn(0x5559f52638a0 msgr2=0x5559f52928a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 shutdown_connections 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 1 -- 172.21.15.12:0/4143653505 wait complete. 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.548+0000 7f8d1bb93740 20 client.5493 trim_cache size 0 max 16384 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker": [ 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.556 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "0" 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "1" 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "2" 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": [ 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.558 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 79, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 65, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 119812, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 34829, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 9, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.008693886, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.002101125, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.009095087, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000729522, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 63, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.038205409, 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000606435 2024-03-08T08:30:31.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 25, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000008340, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000333 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 112432, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 12240 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": [ 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.560 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 37, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 34, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 11690, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 109613, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 7, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 4, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.009656280, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.001427995, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.007240595, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000927119, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 33, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.179458290, 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.005438130 2024-03-08T08:30:31.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 18, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000003244, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000180 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 94688 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": [ 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.562 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 26, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 28, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 6732, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 32301, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 8, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 4, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.007334414, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000431317, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.006244272, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000516752, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 26, 2024-03-08T08:30:31.563 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.141509418, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.005442669 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000226, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000113 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": [ 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.564 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 1, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-MDSRank": [ 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 40, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.003693973, 2024-03-08T08:30:31.565 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000092349 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-PurgeQueue": [ 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000018932, 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000009466 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": [ 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "request": 3, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "reply": 3, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "reply_latency": { 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000621331, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000207110 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_reply": 0, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "forward": 0, 2024-03-08T08:30:31.567 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_complete": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_keys": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_commit": 12, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_split": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_merge": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 10, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_top": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_bottom": 10, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pin_tail": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pinned": 10, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_expired": 0, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_with_caps": 1, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": 2, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "subtrees": 2, 2024-03-08T08:30:31.568 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse": 5, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_hit": 5, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_forward": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_discover": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_dir_fetch": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_remote_ino": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_lock": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "load_cent": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "q": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "exported": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "exported_inodes": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "imported": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "imported_inodes": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_dir_fetch": 0, 2024-03-08T08:30:31.569 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_backtrace_fetch": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_peer_discover": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rfiles": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rbytes": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rsnaps": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_set_tag": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_inotable_repaired": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_inodes": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_file_inodes": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_inode_file_caps": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_revoke": 0, 2024-03-08T08:30:31.570 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_grant": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps_dirty": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_cap_release": 0, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "process_request_cap_release": 1 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_cache": [ 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update_receipt": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_try_discover": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_send_discover": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_handle_discover": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_delayed": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_enqueuing": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_created": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_enqueued": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_reintegrated": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_migrated": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_processing": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_enqueued": 0, 2024-03-08T08:30:31.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_prioritized": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_started": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_completed": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_exportdir": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_flush": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragmentdir": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragstats": 0, 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_inodestats": 0 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics": [ 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.573 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "5277" 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "num_clients": 2 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics-cephfs": [ 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4796", 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.574 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 4, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 2, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 1, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 1, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 1, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 0, 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 0 2024-03-08T08:30:31.575 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.5466", 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 4, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 2, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 0, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 0, 2024-03-08T08:30:31.576 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 0, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 1, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 1, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 1, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 0, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 0 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_log": [ 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.577 INFO:teuthology.orchestra.run.smithi012.stdout: "evlrg": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "evadd": 5, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "evex": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "evtrm": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "ev": 5, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "evexg": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "evexd": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segadd": 2, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segex": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segtrm": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "seg": 2, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segmjr": 2, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segexg": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "segexd": 0, 2024-03-08T08:30:31.578 INFO:teuthology.orchestra.run.smithi012.stdout: "expos": 4194304, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "wrpos": 4196941, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "rdpos": 0, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "jlat": { 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.004504557, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001501519 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "replayed": 0 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_mem": [ 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "ino": 13, 2024-03-08T08:30:31.579 INFO:teuthology.orchestra.run.smithi012.stdout: "ino+": 13, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "ino-": 0, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dir": 12, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dir+": 12, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dir-": 0, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dn": 10, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dn+": 10, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "dn-": 0, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "cap": 2, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "cap+": 2, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "cap-": 0, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "rss": 39616, 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "heap": 215392 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.580 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_server": [ 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_client_request": 3, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_server_request": 0, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_request": 3, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_session": 6, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_peer_request": 0, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "req_create_latency": { 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getattr_latency": { 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-08T08:30:31.581 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000370507, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000185253 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getfilelock_latency": { 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "req_link_latency": { 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookup_latency": { 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.582 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookuphash_latency": { 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupino_latency": { 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupname_latency": { 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.583 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupparent_latency": { 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupsnap_latency": { 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lssnap_latency": { 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mkdir_latency": { 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mknod_latency": { 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mksnap_latency": { 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.585 INFO:teuthology.orchestra.run.smithi012.stdout: "req_open_latency": { 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "req_readdir_latency": { 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rename_latency": { 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "req_renamesnap_latency": { 2024-03-08T08:30:31.586 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "req_snapdiff_latency": { 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmdir_latency": { 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmsnap_latency": { 2024-03-08T08:30:31.587 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmxattr_latency": { 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setattr_latency": { 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000250824, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000250824 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setdirlayout_latency": { 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.588 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setfilelock_latency": { 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setlayout_latency": { 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setxattr_latency": { 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "req_symlink_latency": { 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "req_unlink_latency": { 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_revoke_eviction": 0, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_acquisition_throttle": 0, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getvxattr_latency": { 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_sessions": [ 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "session_count": 2, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "session_add": 2, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "session_remove": 0, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_open": 2, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_stale": 0, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "total_load": 2, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "average_load": 1, 2024-03-08T08:30:31.591 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_session_uptime": 9, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mdthresh_evicted": 0 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": [ 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-08T08:30:31.592 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T08:30:31.593 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T08:30:31.594 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 213755, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 67, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 5312, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 75, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-08T08:30:31.595 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 52240, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 213, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "objecter": [ 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "op_active": 0, 2024-03-08T08:30:31.596 INFO:teuthology.orchestra.run.smithi012.stdout: "op_laggy": 0, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send": 41, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send_bytes": 13757, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_resend": 37, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_reply": 41, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_latency": { 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 41, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.709993775, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.017316921 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_inflight": 0, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "oplen_avg": { 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 41, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 65 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op": 41, 2024-03-08T08:30:31.597 INFO:teuthology.orchestra.run.smithi012.stdout: "op_r": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "op_w": 41, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "op_rmw": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "op_pg": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_stat": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_create": 11, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_read": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_write": 3, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writefull": 7, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writesame": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_append": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_zero": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_truncate": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_delete": 6, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_mapext": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_sparse_read": 0, 2024-03-08T08:30:31.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_clonerange": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_getxattr": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_setxattr": 22, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_cmpxattr": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_rmxattr": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_resetxattrs": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_call": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_watch": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_notify": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls_filter": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_other": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_active": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_send": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_resend": 0, 2024-03-08T08:30:31.599 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_ping": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_active": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_send": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_resend": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_active": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_send": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_resend": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_active": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_send": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_resend": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "command_active": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "command_send": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "command_resend": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "map_epoch": 35, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "map_full": 0, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "map_inc": 1, 2024-03-08T08:30:31.600 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_sessions": 6, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_open": 6, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_close": 0, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_laggy": 0, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_wr": 15, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_rd": 0, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_del": 1 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "oft": [ 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_objs": 1, 2024-03-08T08:30:31.601 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_kv_pairs": 0, 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_updates": 0, 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_removes": 0 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "osd-slow-ops": [ 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_ops_count": 0 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: "purge_queue": [ 2024-03-08T08:30:31.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_high_water": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed_ops": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed": 0, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_item_in_journal": 0 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": [ 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.603 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 134, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 120907, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 134, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 134, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 120907, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_bytes": [ 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 41, 2024-03-08T08:30:31.605 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 13757, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 41, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 13757, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_ops": [ 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.606 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 1024, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 41, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 41, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 41, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 41, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle": [ 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:31.608 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 0, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 0, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle-0x559bb8a9a6e0": [ 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:31.609 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 5, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 2637, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 5, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 3, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 2637, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.611 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.611 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-08T08:30:31.611 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 msgr2=0x7f8cf406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 0x7f8cf406d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d140b8b30 tx=0x7f8cfc009040 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 msgr2=0x7f8d141f8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0019d0 tx=0x7f8d0c0074f0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 shutdown_connections 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d140b56c0 0x7f8d140b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d1415f590 0x7f8d141f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 --2- 172.21.15.12:0/3922281644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d00004d40 0x7f8d141f8dc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 >> 172.21.15.12:0/3922281644 conn(0x7f8d1406c240 msgr2=0x7f8d140ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 shutdown_connections 2024-03-08T08:30:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31.549+0000 7f8d18ed4640 1 -- 172.21.15.12:0/3922281644 wait complete. 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-03-08T08:30:31.613 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-03-08T08:30:31.614 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-03-08T08:30:31.615 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-03-08T08:30:31.615 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:30:31.615 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:31.615 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-03-08T08:30:31.615 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-08T08:30:31.695 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.51735.asok 2024-03-08T08:30:31.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:30:31.706 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51735.asok 2024-03-08T08:30:31.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:31.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51735.asok mds_sessions 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5466, 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5466 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:31.845 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1928274948 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.5466 v1:192.168.0.1:0/1928274948", 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "v1:192.168.0.1:0/1928274948", 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions": [ 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": 0, 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "addrs": { 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6836", 2024-03-08T08:30:31.846 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2820296658 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6837", 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2820296658 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "seq": 0, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_gen": 0, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_ttl": "2024-03-08T08:31:20.506985+0000", 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "last_cap_renew_request": "2024-03-08T08:30:20.506985+0000", 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_renew_seq": 2, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "num_caps": 1, 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "open" 2024-03-08T08:30:31.847 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:31.848 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:31.848 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 15 2024-03-08T08:30:31.848 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:31.853 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-08T08:30:31.853 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:31.854 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2024-03-08T08:30:31.855 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:31.947 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.43673.asok 2024-03-08T08:30:31.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:30:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:30:31.959 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43673.asok 2024-03-08T08:30:31.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:30:31.959 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43673.asok mds_sessions 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4796, 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4796 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-08T08:30:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2054950743 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4796 v1:192.168.0.1:0/2054950743", 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "v1:192.168.0.1:0/2054950743", 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions": [ 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": 0, 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "addrs": { 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "addrvec": [ 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v2", 2024-03-08T08:30:32.098 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.12:6836", 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2820296658 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.12:6837", 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2820296658 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "seq": 0, 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_gen": 0, 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_ttl": "2024-03-08T08:31:22.025982+0000", 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "last_cap_renew_request": "2024-03-08T08:30:22.025982+0000", 2024-03-08T08:30:32.099 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_renew_seq": 2, 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout: "num_caps": 1, 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "open" 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 15 2024-03-08T08:30:32.100 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-08T08:30:32.106 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:30:32.106 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:32.120 DEBUG:teuthology.orchestra.run.smithi012:> stdin-killer -- python3 -c ' 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> import uuid 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> n = 1000 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> create_hard_links = False 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> if 0 > 0: 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> create_hard_links = True 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> path = "/home/ubuntu/cephtest/mnt.0/test_dir/test_file" 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> dpath = os.path.dirname(path) 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> fnameprefix = os.path.basename(path) 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> os.makedirs(dpath, exist_ok=True) 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> try: 2024-03-08T08:30:32.121 DEBUG:teuthology.orchestra.run.smithi012:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> for i in range(n): 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> with open(fpath, '"'"'w'"'"') as f: 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> f.write(f"{i}") 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> if True: 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> f.flush() 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> os.fsync(f.fileno()) 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> if False: 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> os.unlink(fpath) 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> if False: 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> os.fsync(dirfd) 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> if create_hard_links: 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> for j in range(0): 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-08T08:30:32.122 DEBUG:teuthology.orchestra.run.smithi012:> if False: 2024-03-08T08:30:32.123 DEBUG:teuthology.orchestra.run.smithi012:> os.fsync(dirfd) 2024-03-08T08:30:32.123 DEBUG:teuthology.orchestra.run.smithi012:> finally: 2024-03-08T08:30:32.123 DEBUG:teuthology.orchestra.run.smithi012:> os.close(dirfd) 2024-03-08T08:30:32.123 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-08T08:30:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:30:35.969 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T08:30:35.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 counter dump 2024-03-08T08:30:36.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.111+0000 7f0e6082e640 1 Processor -- start 2024-03-08T08:30:36.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.111+0000 7f0e6082e640 1 -- start start 2024-03-08T08:30:36.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e6082e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 0x7f0e5c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e6082e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0e5c0ce480 0x7f0e5c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e6082e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0cd020 con 0x7f0e5c144db0 2024-03-08T08:30:36.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e6082e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c0cd300 con 0x7f0e5c0ce480 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e6082e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c0cd5e0 con 0x7f0e5c0af320 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e5a59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0e5c144db0 0x7f0e5c0cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46618/0 (socket says 172.21.15.12:46618) 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e5a59c640 1 -- 172.21.15.12:0/1013853444 learned_addr learned my addr 172.21.15.12:0/1013853444 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/1013853444 >> v2:172.21.15.12:3300/0 conn(0x7f0e5c0ce480 0x7f0e5c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.112+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/1013853444 >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 0x7f0e5c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/1013853444 >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 msgr2=0x7f0e5c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/1013853444 >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 0x7f0e5c06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/1013853444 >> v1:172.21.15.12:6789/0 conn(0x7f0e5c144db0 legacy=0x7f0e5c0cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/1013853444 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0cd8c0 con 0x7f0e5c0ce480 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/1013853444 >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 0x7f0e5c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/1013853444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c06dd40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0b2850 tx=0x7f0e5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e28a1036241a69 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5959a640 1 -- 172.21.15.12:0/1013853444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54039070 con 0x7f0e5c0ce480 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5959a640 1 -- 172.21.15.12:0/1013853444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5402cc80 con 0x7f0e5c0ce480 2024-03-08T08:30:36.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.113+0000 7f0e5959a640 1 -- 172.21.15.12:0/1013853444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54034660 con 0x7f0e5c0ce480 2024-03-08T08:30:36.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 -- 172.21.15.12:0/1013853444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 msgr2=0x7f0e5c06dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 --2- 172.21.15.12:0/1013853444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c06dd40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0b2850 tx=0x7f0e5402bed0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 -- 172.21.15.12:0/1013853444 shutdown_connections 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 --2- 172.21.15.12:0/1013853444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c06dd40 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 --2- 172.21.15.12:0/1013853444 >> v2:172.21.15.156:3300/0 conn(0x7f0e5c0af320 0x7f0e5c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 -- 172.21.15.12:0/1013853444 >> 172.21.15.12:0/1013853444 conn(0x7f0e5c06abd0 msgr2=0x7f0e5c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 -- 172.21.15.12:0/1013853444 shutdown_connections 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.114+0000 7f0e6082e640 1 -- 172.21.15.12:0/1013853444 wait complete. 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.115+0000 7f0e6082e640 1 Processor -- start 2024-03-08T08:30:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.115+0000 7f0e6082e640 1 -- start start 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 0x7f0e5c13b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e5c144db0 0x7f0e5c15e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c06b840 con 0x7f0e5c0ce480 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c06b6c0 con 0x7f0e5c0af320 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e6082e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c145430 con 0x7f0e5c144db0 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e59d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e59d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47908/0 (socket says 172.21.15.12:47908) 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e59d9b640 1 -- 172.21.15.12:0/90091437 learned_addr learned my addr 172.21.15.12:0/90091437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e5c144db0 0x7f0e5c15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.116+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 0x7f0e5c13b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 -- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e5c144db0 msgr2=0x7f0e5c15e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e5c144db0 0x7f0e5c15e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 -- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 msgr2=0x7f0e5c13b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 0x7f0e5c13b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 -- 172.21.15.12:0/90091437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c158340 con 0x7f0e5c0ce480 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 0x7f0e5c13b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e7e0 tx=0x7f0e4c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e52ffd640 1 -- 172.21.15.12:0/90091437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c00f040 con 0x7f0e5c0ce480 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e52ffd640 1 -- 172.21.15.12:0/90091437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c004030 con 0x7f0e5c0ce480 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c158620 con 0x7f0e5c0ce480 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e52ffd640 1 -- 172.21.15.12:0/90091437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c004400 con 0x7f0e5c0ce480 2024-03-08T08:30:36.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.117+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c158be0 con 0x7f0e5c0ce480 2024-03-08T08:30:36.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.118+0000 7f0e52ffd640 1 -- 172.21.15.12:0/90091437 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e4c00c6d0 con 0x7f0e5c0ce480 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.119+0000 7f0e52ffd640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.119+0000 7f0e52ffd640 1 -- 172.21.15.12:0/90091437 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e4c089f10 con 0x7f0e5c0ce480 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.119+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.119+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e44000f60 tx=0x7f0e44009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.120+0000 7f0e63415740 1 Processor -- start 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.120+0000 7f0e63415740 1 -- start start 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 0x559d94c870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94ca7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d94bc3e30 con 0x559d94ca67e0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d94b6cd20 con 0x559d94ca70c0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d94bbfed0 con 0x559d94c86cf0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47922/0 (socket says 172.21.15.12:47922) 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/3260095809 learned_addr learned my addr 172.21.15.12:0/3260095809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94ca7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 0x559d94c870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 msgr2=0x559d94c870d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 0x559d94c870d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 msgr2=0x559d94ca7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94ca7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/3260095809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d948d83f0 con 0x559d94ca67e0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94ca7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 0x559d94c870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009e50 tx=0x7f0e5402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b012e5f39124d4af server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.121+0000 7f0e3bfff640 1 -- 172.21.15.12:0/3260095809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54045070 con 0x559d94ca67e0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e3bfff640 1 -- 172.21.15.12:0/3260095809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5403a040 con 0x559d94ca67e0 2024-03-08T08:30:36.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e3bfff640 1 -- 172.21.15.12:0/3260095809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54039040 con 0x559d94ca67e0 2024-03-08T08:30:36.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 -- 172.21.15.12:0/3260095809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 msgr2=0x559d94c596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009e50 tx=0x7f0e5402e000 comp rx=0 tx=0).stop 2024-03-08T08:30:36.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 -- 172.21.15.12:0/3260095809 shutdown_connections 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c596b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94ca7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 --2- 172.21.15.12:0/3260095809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c86cf0 0x559d94c870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 -- 172.21.15.12:0/3260095809 >> 172.21.15.12:0/3260095809 conn(0x559d94c5b920 msgr2=0x559d94c5d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 -- 172.21.15.12:0/3260095809 shutdown_connections 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.122+0000 7f0e63415740 1 -- 172.21.15.12:0/3260095809 wait complete. 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.123+0000 7f0e63415740 1 -- start start 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.123+0000 7f0e63415740 1 Processor -- start 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c5f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c7cb50 0x559d94c80980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d94c8a040 con 0x559d94ca67e0 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d94b62390 con 0x559d94ca70c0 2024-03-08T08:30:36.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d94ca5a60 con 0x559d94c7cb50 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e63415740 10 client.0 _ll_register_callbacks cb 0x559d94b7fc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47928/0 (socket says 172.21.15.12:47928) 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3462797141 learned_addr learned my addr 172.21.15.12:0/3462797141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c5f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c7cb50 0x559d94c80980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c7cb50 msgr2=0x559d94c80980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c7cb50 0x559d94c80980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 msgr2=0x559d94c5f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c5f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3462797141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d949a27d0 con 0x559d94ca67e0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007170 tx=0x7f0e44000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbfad89bb36366b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.124+0000 7f0e397fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440076f0 con 0x559d94ca67e0 2024-03-08T08:30:36.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e4400ecc0 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e5403a040 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e44022430 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 10 client.5508 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x559d94c5eb90 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5403a040 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0e440225d0 con 0x559d94ca67e0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 10 client.5508 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 10 client.5508 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 20 client.5508 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94c80ec0 0x559d94c85370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 4 client.5508 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d94bd1370 con 0x559d94c80ec0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.125+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94c80ec0 0x559d94c85370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e397fa640 10 client.5508 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:36.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94c80ec0 0x559d94c85370 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659b133d39ad7719 server_cookie=7a7cdbc94c916aa2 in_seq=0 out_seq=0 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e397fa640 1 -- 172.21.15.12:0/3462797141 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d94bd1370 con 0x559d94c80ec0 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e397fa640 10 client.5508 handle_command_reply: tid=0 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 client.5508 shutdown 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 msgr2=0x559d94c7ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007170 tx=0x7f0e44000bc0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 shutdown_connections 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94c80ec0 0x559d94c85370 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94c7cb50 0x559d94c80980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c5f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e397fa640 0 client.5508 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 --2- 172.21.15.12:0/3462797141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca67e0 0x559d94c7ba80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 >> 172.21.15.12:0/3462797141 conn(0x559d94c86cf0 msgr2=0x559d94c585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 shutdown_connections 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 1 -- 172.21.15.12:0/3462797141 wait complete. 2024-03-08T08:30:36.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.126+0000 7f0e63415740 20 client.5508 trim_cache size 0 max 16384 2024-03-08T08:30:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.138+0000 7f0e63415740 1 Processor -- start 2024-03-08T08:30:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.138+0000 7f0e63415740 1 -- start start 2024-03-08T08:30:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 0x559d94c7fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1fba80 con 0x559d94ca70c0 2024-03-08T08:30:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b41f0 con 0x559d94c7cb50 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f2050 con 0x559d94ca67e0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47930/0 (socket says 172.21.15.12:47930) 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/2956016246 learned_addr learned my addr 172.21.15.12:0/2956016246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 0x559d94c7fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 msgr2=0x559d94c59670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 msgr2=0x559d94c7fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 0x559d94c7fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 -- 172.21.15.12:0/2956016246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d94c585a0 con 0x559d94ca70c0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 0x559d94c7fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.139+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402bf10 tx=0x7f0e54031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b457a5baff141a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e3b7fe640 1 -- 172.21.15.12:0/2956016246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54043050 con 0x559d94ca70c0 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e3b7fe640 1 -- 172.21.15.12:0/2956016246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5404a650 con 0x559d94ca70c0 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e3b7fe640 1 -- 172.21.15.12:0/2956016246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54039040 con 0x559d94ca70c0 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 -- 172.21.15.12:0/2956016246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 msgr2=0x559d94c59bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402bf10 tx=0x7f0e54031000 comp rx=0 tx=0).stop 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 -- 172.21.15.12:0/2956016246 shutdown_connections 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94ca70c0 0x559d94c59bb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 --2- 172.21.15.12:0/2956016246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94c7cb50 0x559d94c7fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 -- 172.21.15.12:0/2956016246 >> 172.21.15.12:0/2956016246 conn(0x559d94c86cf0 msgr2=0x559d94c7b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 -- 172.21.15.12:0/2956016246 shutdown_connections 2024-03-08T08:30:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.140+0000 7f0e63415740 1 -- 172.21.15.12:0/2956016246 wait complete. 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.141+0000 7f0e63415740 1 -- start start 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.141+0000 7f0e63415740 1 Processor -- start 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c59980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d94c85dd0 con 0x559d94c7cb50 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f2650 con 0x559d94ca70c0 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d94c859d0 con 0x559d94ca67e0 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 10 client.0 _ll_register_callbacks cb 0x7f0e44001320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47944/0 (socket says 172.21.15.12:47944) 2024-03-08T08:30:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3566807260 learned_addr learned my addr 172.21.15.12:0/3566807260 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5ad9d640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c59980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 msgr2=0x559d94c59440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 msgr2=0x559d94c59980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c59980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 -- 172.21.15.12:0/3566807260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d94c7d5f0 con 0x559d94c7cb50 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e5a59c640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e160 tx=0x7f0e44007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685acaea201618ed server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e397fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44030040 con 0x559d94c7cb50 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e44022b20 con 0x559d94c7cb50 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e5404a650 con 0x559d94c7cb50 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e4403d050 con 0x559d94c7cb50 2024-03-08T08:30:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 10 client.5514 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.142+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x559d94d136a0 con 0x559d94c7cb50 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5404a650 con 0x559d94c7cb50 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0e44040020 con 0x559d94c7cb50 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 10 client.5514 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 10 client.5514 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 20 client.5514 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94cb0510 0x559d94cc67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 4 client.5514 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "counter dump"}] 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "counter dump"}) v1 -- 0x559d94bcb6c0 con 0x559d94cb0510 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94cb0510 0x559d94cc67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e397fa640 10 client.5514 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.143+0000 7f0e59d9b640 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94cb0510 0x559d94cc67b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9551d4ca898d86be server_cookie=b2e4c8a9c8d3c3d5 in_seq=0 out_seq=0 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e397fa640 1 -- 172.21.15.12:0/3566807260 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26919 (crc 0 0 0) 0x559d94bcb6c0 con 0x559d94cb0510 2024-03-08T08:30:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e397fa640 10 client.5514 handle_command_reply: tid=0 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 client.5514 shutdown 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 msgr2=0x559d94d13160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e160 tx=0x7f0e44007000 comp rx=0 tx=0).stop 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 shutdown_connections 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x559d94cb0510 0x559d94cc67b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d94ca70c0 0x559d94c59980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x559d94ca67e0 0x559d94c59440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e397fa640 0 client.5514 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 --2- 172.21.15.12:0/3566807260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d94c7cb50 0x559d94d13160 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 >> 172.21.15.12:0/3566807260 conn(0x559d94c86cf0 msgr2=0x559d94c54cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 shutdown_connections 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 1 -- 172.21.15.12:0/3566807260 wait complete. 2024-03-08T08:30:36.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.144+0000 7f0e63415740 20 client.5514 trim_cache size 0 max 16384 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker": [ 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "0" 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "1" 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "2" 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:36.153 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": [ 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 3087, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 4069, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 937628, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 1778131, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 9, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.196058820, 2024-03-08T08:30:36.154 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.095655374, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.149773851, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000882106, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 4067, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.082812091, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000020361 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 7030, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.001168650, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000166 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 112704, 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 12864 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": [ 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 43, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 36, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 14130, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 111755, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 7, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 4, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.010147022, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.001589063, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.007393730, 2024-03-08T08:30:36.156 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.001098556, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 35, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.179491933, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.005128340 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 25, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000004142, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000165 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 96480 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.157 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": [ 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1034, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 1036, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 254684, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 4408985, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 10, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 6, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.064121185, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.022864817, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.018546101, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.018467960, 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:36.158 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1034, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.156563775, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000151415 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 4, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000397, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000099 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": [ 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.159 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 1, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-MDSRank": [ 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2047, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.436268498, 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000213125 2024-03-08T08:30:36.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-PurgeQueue": [ 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000018932, 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000009466 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.161 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": [ 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "request": 1005, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "reply": 1005, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "reply_latency": { 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1005, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.507183258, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000504659 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_reply": 0, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "forward": 0, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_complete": 0, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_keys": 0, 2024-03-08T08:30:36.162 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_commit": 12, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_split": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_merge": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 1011, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_top": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_bottom": 1011, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pin_tail": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pinned": 1011, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_expired": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_with_caps": 1002, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": 1003, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "subtrees": 2, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse": 1010, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_hit": 1008, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_forward": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_discover": 0, 2024-03-08T08:30:36.163 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_dir_fetch": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_remote_ino": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_lock": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "load_cent": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "q": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "exported": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "exported_inodes": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "imported": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "imported_inodes": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_dir_fetch": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_backtrace_fetch": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_peer_discover": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rfiles": 808, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rbytes": 2314, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rsnaps": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T08:30:36.164 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_set_tag": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_inotable_repaired": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_inodes": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_file_inodes": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_inode_file_caps": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_revoke": 2, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_grant": 1000, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps": 1002, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps_dirty": 1000, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_cap_release": 0, 2024-03-08T08:30:36.165 INFO:teuthology.orchestra.run.smithi012.stdout: "process_request_cap_release": 1002 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_cache": [ 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update_receipt": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_try_discover": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_send_discover": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_handle_discover": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_delayed": 0, 2024-03-08T08:30:36.166 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_enqueuing": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_created": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_enqueued": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_reintegrated": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_migrated": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_processing": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_enqueued": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_prioritized": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_started": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_completed": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_exportdir": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_flush": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragmentdir": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragstats": 0, 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_inodestats": 0 2024-03-08T08:30:36.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics": [ 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "5277" 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "num_clients": 2 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics-cephfs": [ 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.168 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4796", 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 4, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 2, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 0, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 0, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 0, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 1, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 1, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 1, 2024-03-08T08:30:36.169 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 0, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 0 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.5466", 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 6170, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 3, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:36.170 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 617, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 0, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 1, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 618, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 618, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 618, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 616, 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 1738 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_log": [ 2024-03-08T08:30:36.171 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evlrg": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evadd": 2007, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evex": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evtrm": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "ev": 2007, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evexg": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "evexd": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segadd": 3, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segex": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segtrm": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "seg": 3, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segmjr": 2, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segexg": 0, 2024-03-08T08:30:36.172 INFO:teuthology.orchestra.run.smithi012.stdout: "segexd": 0, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "expos": 4194304, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "wrpos": 8143572, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "rdpos": 0, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "jlat": { 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1004, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 1.335903486, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001330581 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "replayed": 0 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_mem": [ 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.173 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "ino": 1014, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "ino+": 1014, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "ino-": 0, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dir": 13, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dir+": 13, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dir-": 0, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dn": 1011, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dn+": 1011, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "dn-": 0, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "cap": 1003, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "cap+": 1003, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "cap-": 0, 2024-03-08T08:30:36.174 INFO:teuthology.orchestra.run.smithi012.stdout: "rss": 53824, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "heap": 215392 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_server": [ 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_client_request": 1007, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_server_request": 0, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_request": 1005, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_session": 1006, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_peer_request": 0, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "req_create_latency": { 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1000, 2024-03-08T08:30:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.503472675, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000503472 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getattr_latency": { 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000601778, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000200592 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getfilelock_latency": { 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "req_link_latency": { 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.176 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookup_latency": { 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookuphash_latency": { 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupino_latency": { 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.177 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupname_latency": { 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupparent_latency": { 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupsnap_latency": { 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lssnap_latency": { 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mkdir_latency": { 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.002857981, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.002857981 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mknod_latency": { 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.179 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mksnap_latency": { 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "req_open_latency": { 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "req_readdir_latency": { 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rename_latency": { 2024-03-08T08:30:36.180 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "req_renamesnap_latency": { 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "req_snapdiff_latency": { 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmdir_latency": { 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmsnap_latency": { 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmxattr_latency": { 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setattr_latency": { 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000250824, 2024-03-08T08:30:36.182 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000250824 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setdirlayout_latency": { 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setfilelock_latency": { 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setlayout_latency": { 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setxattr_latency": { 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "req_symlink_latency": { 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "req_unlink_latency": { 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_revoke_eviction": 0, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_acquisition_throttle": 0, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getvxattr_latency": { 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_sessions": [ 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "session_count": 2, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "session_add": 2, 2024-03-08T08:30:36.185 INFO:teuthology.orchestra.run.smithi012.stdout: "session_remove": 0, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_open": 2, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_stale": 0, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "total_load": 983, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "average_load": 491, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_session_uptime": 14, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "mdthresh_evicted": 0 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": [ 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-08T08:30:36.186 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-08T08:30:36.187 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T08:30:36.188 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 230590, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 84, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 5312, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 75, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T08:30:36.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 2328672, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 15034, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "objecter": [ 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.190 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_active": 0, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_laggy": 0, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send": 1047, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send_bytes": 4024357, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_resend": 37, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_reply": 1047, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_latency": { 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1047, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 2.045982043, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001954137 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "op_inflight": 0, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "oplen_avg": { 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1047, 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 1075 2024-03-08T08:30:36.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "op": 1047, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "op_r": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "op_w": 1047, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "op_rmw": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "op_pg": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_stat": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_create": 11, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_read": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_write": 1004, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writefull": 8, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writesame": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_append": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_zero": 0, 2024-03-08T08:30:36.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_truncate": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_delete": 6, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_mapext": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_sparse_read": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_clonerange": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_getxattr": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_setxattr": 22, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_cmpxattr": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_rmxattr": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_resetxattrs": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_call": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_watch": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_notify": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls_filter": 0, 2024-03-08T08:30:36.193 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_other": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_active": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_send": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_resend": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_ping": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_active": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_send": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_resend": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_active": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_send": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_resend": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_active": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_send": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_resend": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "command_active": 0, 2024-03-08T08:30:36.194 INFO:teuthology.orchestra.run.smithi012.stdout: "command_send": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "command_resend": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "map_epoch": 35, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "map_full": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "map_inc": 1, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_sessions": 6, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_open": 6, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_close": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_laggy": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_wr": 23, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_rd": 0, 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_del": 1 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: "oft": [ 2024-03-08T08:30:36.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_objs": 1, 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_kv_pairs": 0, 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_updates": 0, 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_removes": 0 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "osd-slow-ops": [ 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_ops_count": 0 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "purge_queue": [ 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_high_water": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed_ops": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed": 0, 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_item_in_journal": 0 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.197 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": [ 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 4156, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 827083, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 4156, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 4156, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 827083, 2024-03-08T08:30:36.198 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_bytes": [ 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:36.199 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 1047, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 4024357, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1047, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 4024357, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_ops": [ 2024-03-08T08:30:36.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 1024, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 1047, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 1047, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1047, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 1047, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.201 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle": [ 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:36.202 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle-0x559bb8a9a6e0": [ 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:36.203 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 2007, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 3949268, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 2007, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1004, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 3949268, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:36.204 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:36.205 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-08T08:30:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 msgr2=0x7f0e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e44000f60 tx=0x7f0e44009040 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 msgr2=0x7f0e5c1e43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e7e0 tx=0x7f0e4c00eca0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 shutdown_connections 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0e5c144db0 0x7f0e5c15e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e5c0ce480 0x7f0e5c1e43b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 --2- 172.21.15.12:0/90091437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e5c0af320 0x7f0e5c13b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:36.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 >> 172.21.15.12:0/90091437 conn(0x7f0e5c06abd0 msgr2=0x7f0e5c0cdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:36.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 shutdown_connections 2024-03-08T08:30:36.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:36.145+0000 7f0e6082e640 1 -- 172.21.15.12:0/90091437 wait complete. 2024-03-08T08:30:36.207 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.207 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.223 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.223 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.223 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.223 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_0", '"'"'r'"'"') as f: 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.224 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.266 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_0 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_0 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511627777 Links: 1 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.237279046 +0000 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.240535481 +0000 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.240535481 +0000 2024-03-08T08:30:36.288 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.288 INFO:tasks.cephfs.mount:File test_dir/test_file_0 became visible from 1 after 0s 2024-03-08T08:30:36.289 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.289 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.330 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.343 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_1", '"'"'r'"'"') as f: 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.344 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.386 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_1 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_1 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628278 Links: 1 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.243201043 +0000 2024-03-08T08:30:36.405 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.244474368 +0000 2024-03-08T08:30:36.406 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.244474368 +0000 2024-03-08T08:30:36.406 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.406 INFO:tasks.cephfs.mount:File test_dir/test_file_1 became visible from 1 after 0s 2024-03-08T08:30:36.406 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.406 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.449 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_2", '"'"'r'"'"') as f: 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.461 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.462 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.462 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.503 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_2 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_2 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628279 Links: 1 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.247461390 +0000 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.248472908 +0000 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.248472908 +0000 2024-03-08T08:30:36.522 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.523 INFO:tasks.cephfs.mount:File test_dir/test_file_2 became visible from 1 after 0s 2024-03-08T08:30:36.523 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.523 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.567 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.577 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.577 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.577 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_3", '"'"'r'"'"') as f: 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.578 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.620 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_3 2024-03-08T08:30:36.639 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_3 2024-03-08T08:30:36.639 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628280 Links: 1 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.251169462 +0000 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.252149136 +0000 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.252149136 +0000 2024-03-08T08:30:36.640 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.640 INFO:tasks.cephfs.mount:File test_dir/test_file_3 became visible from 1 after 0s 2024-03-08T08:30:36.640 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.640 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.684 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.695 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_4", '"'"'r'"'"') as f: 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.696 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.737 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_4 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_4 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628281 Links: 1 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.254452467 +0000 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.255449289 +0000 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.255449289 +0000 2024-03-08T08:30:36.757 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.758 INFO:tasks.cephfs.mount:File test_dir/test_file_4 became visible from 1 after 0s 2024-03-08T08:30:36.758 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.758 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.803 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_5", '"'"'r'"'"') as f: 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.813 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.814 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.814 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.814 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.814 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.814 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.855 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_5 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_5 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628282 Links: 1 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.258013847 +0000 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.258994456 +0000 2024-03-08T08:30:36.875 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.258994456 +0000 2024-03-08T08:30:36.876 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.876 INFO:tasks.cephfs.mount:File test_dir/test_file_5 became visible from 1 after 0s 2024-03-08T08:30:36.876 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.876 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:36.919 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:36.930 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_6", '"'"'r'"'"') as f: 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:36.931 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:36.932 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:36.973 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_6 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_6 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628283 Links: 1 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.261500024 +0000 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.262499868 +0000 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.262499868 +0000 2024-03-08T08:30:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:36.993 INFO:tasks.cephfs.mount:File test_dir/test_file_6 became visible from 1 after 0s 2024-03-08T08:30:36.994 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:36.994 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.037 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.048 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.048 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_7", '"'"'r'"'"') as f: 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.049 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.091 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_7 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_7 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628284 Links: 1 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.265053371 +0000 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.266076044 +0000 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.266076044 +0000 2024-03-08T08:30:37.111 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.111 INFO:tasks.cephfs.mount:File test_dir/test_file_7 became visible from 1 after 0s 2024-03-08T08:30:37.112 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.112 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.155 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.167 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.167 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_8", '"'"'r'"'"') as f: 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.168 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.210 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_8 2024-03-08T08:30:37.229 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_8 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628285 Links: 1 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.268812654 +0000 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.269849119 +0000 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.269849119 +0000 2024-03-08T08:30:37.230 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.230 INFO:tasks.cephfs.mount:File test_dir/test_file_8 became visible from 1 after 0s 2024-03-08T08:30:37.230 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.230 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.274 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_9", '"'"'r'"'"') as f: 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.285 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.286 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.286 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.327 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_9 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_9 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 1 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628286 Links: 1 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.272592994 +0000 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.273900915 +0000 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.273900915 +0000 2024-03-08T08:30:37.347 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.348 INFO:tasks.cephfs.mount:File test_dir/test_file_9 became visible from 1 after 0s 2024-03-08T08:30:37.348 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.348 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.391 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.402 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_10", '"'"'r'"'"') as f: 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.403 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.404 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.445 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_10 2024-03-08T08:30:37.464 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_10 2024-03-08T08:30:37.464 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628287 Links: 1 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.276613491 +0000 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.277857662 +0000 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.277857662 +0000 2024-03-08T08:30:37.465 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.465 INFO:tasks.cephfs.mount:File test_dir/test_file_10 became visible from 1 after 0s 2024-03-08T08:30:37.465 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.465 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.509 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_11", '"'"'r'"'"') as f: 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.521 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.563 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_11 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_11 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628288 Links: 1 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.280477185 +0000 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.281744054 +0000 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.281744054 +0000 2024-03-08T08:30:37.582 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.582 INFO:tasks.cephfs.mount:File test_dir/test_file_11 became visible from 1 after 0s 2024-03-08T08:30:37.582 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.583 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.627 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_12", '"'"'r'"'"') as f: 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.637 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.638 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.638 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.638 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.638 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.679 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_12 2024-03-08T08:30:37.694 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_12 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628289 Links: 1 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.284561944 +0000 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.285908744 +0000 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.285908744 +0000 2024-03-08T08:30:37.695 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.695 INFO:tasks.cephfs.mount:File test_dir/test_file_12 became visible from 1 after 0s 2024-03-08T08:30:37.695 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.695 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.743 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_13", '"'"'r'"'"') as f: 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.750 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.751 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.751 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.751 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.751 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.751 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.792 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_13 2024-03-08T08:30:37.807 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_13 2024-03-08T08:30:37.807 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.807 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628290 Links: 1 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.288616736 +0000 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.289852817 +0000 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.289852817 +0000 2024-03-08T08:30:37.808 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.808 INFO:tasks.cephfs.mount:File test_dir/test_file_13 became visible from 1 after 0s 2024-03-08T08:30:37.808 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.808 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.855 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.862 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.862 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.862 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.862 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_14", '"'"'r'"'"') as f: 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.863 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:37.905 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_14 2024-03-08T08:30:37.920 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_14 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628291 Links: 1 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.292510335 +0000 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.293711945 +0000 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.293711945 +0000 2024-03-08T08:30:37.921 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:37.921 INFO:tasks.cephfs.mount:File test_dir/test_file_14 became visible from 1 after 0s 2024-03-08T08:30:37.921 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:37.921 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:37.969 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:37.975 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_15", '"'"'r'"'"') as f: 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:37.976 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:37.977 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.018 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_15 2024-03-08T08:30:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_15 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628292 Links: 1 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.296595026 +0000 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.297624668 +0000 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.297624668 +0000 2024-03-08T08:30:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.034 INFO:tasks.cephfs.mount:File test_dir/test_file_15 became visible from 1 after 0s 2024-03-08T08:30:38.034 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.034 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_16", '"'"'r'"'"') as f: 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.089 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.090 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.131 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_16 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_16 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628293 Links: 1 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.300540918 +0000 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.301621520 +0000 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.301621520 +0000 2024-03-08T08:30:38.147 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.147 INFO:tasks.cephfs.mount:File test_dir/test_file_16 became visible from 1 after 0s 2024-03-08T08:30:38.148 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.148 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.201 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_17", '"'"'r'"'"') as f: 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.202 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.203 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.203 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.244 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_17 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_17 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628294 Links: 1 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.304188820 +0000 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.305125535 +0000 2024-03-08T08:30:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.305125535 +0000 2024-03-08T08:30:38.261 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.261 INFO:tasks.cephfs.mount:File test_dir/test_file_17 became visible from 1 after 0s 2024-03-08T08:30:38.261 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.261 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.307 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.316 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_18", '"'"'r'"'"') as f: 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.317 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.318 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.334 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_18 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_18 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628295 Links: 1 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.350 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.307879574 +0000 2024-03-08T08:30:38.351 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.308861047 +0000 2024-03-08T08:30:38.351 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.308861047 +0000 2024-03-08T08:30:38.351 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.351 INFO:tasks.cephfs.mount:File test_dir/test_file_18 became visible from 1 after 0s 2024-03-08T08:30:38.351 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.351 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.398 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.406 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_19", '"'"'r'"'"') as f: 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.407 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.448 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_19 2024-03-08T08:30:38.451 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_19 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628296 Links: 1 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.311528892 +0000 2024-03-08T08:30:38.464 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.312494605 +0000 2024-03-08T08:30:38.465 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.312494605 +0000 2024-03-08T08:30:38.465 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.465 INFO:tasks.cephfs.mount:File test_dir/test_file_19 became visible from 1 after 0s 2024-03-08T08:30:38.465 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.465 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.512 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_20", '"'"'r'"'"') as f: 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.520 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.521 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.562 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_20 2024-03-08T08:30:38.569 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_20 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628297 Links: 1 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.314902156 +0000 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.315851721 +0000 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.315851721 +0000 2024-03-08T08:30:38.579 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.579 INFO:tasks.cephfs.mount:File test_dir/test_file_20 became visible from 1 after 0s 2024-03-08T08:30:38.580 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.580 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.627 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.633 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.633 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.633 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_21", '"'"'r'"'"') as f: 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.634 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.676 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_21 2024-03-08T08:30:38.685 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_21 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628298 Links: 1 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.691 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.318296955 +0000 2024-03-08T08:30:38.692 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.319220664 +0000 2024-03-08T08:30:38.692 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.319220664 +0000 2024-03-08T08:30:38.692 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.692 INFO:tasks.cephfs.mount:File test_dir/test_file_21 became visible from 1 after 0s 2024-03-08T08:30:38.692 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.692 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.740 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.746 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.746 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_22", '"'"'r'"'"') as f: 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.747 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.789 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_22 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_22 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628299 Links: 1 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.322009949 +0000 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.322988848 +0000 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.322988848 +0000 2024-03-08T08:30:38.805 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.806 INFO:tasks.cephfs.mount:File test_dir/test_file_22 became visible from 1 after 0s 2024-03-08T08:30:38.806 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.806 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.806 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.851 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_23", '"'"'r'"'"') as f: 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.861 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.862 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.862 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.862 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.862 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.862 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.903 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_23 2024-03-08T08:30:38.918 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_23 2024-03-08T08:30:38.918 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.918 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628300 Links: 1 2024-03-08T08:30:38.918 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.919 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.919 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.325320882 +0000 2024-03-08T08:30:38.919 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.326244209 +0000 2024-03-08T08:30:38.919 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.326244209 +0000 2024-03-08T08:30:38.919 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.919 INFO:tasks.cephfs.mount:File test_dir/test_file_23 became visible from 1 after 0s 2024-03-08T08:30:38.919 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.919 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:38.920 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_24", '"'"'r'"'"') as f: 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:38.935 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:38.936 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:38.936 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:38.967 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:38.968 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_24 2024-03-08T08:30:38.984 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_24 2024-03-08T08:30:38.984 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:38.984 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628301 Links: 1 2024-03-08T08:30:38.984 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:38.985 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:38.985 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.328630910 +0000 2024-03-08T08:30:38.985 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.329569720 +0000 2024-03-08T08:30:38.985 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.329569720 +0000 2024-03-08T08:30:38.985 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:38.985 INFO:tasks.cephfs.mount:File test_dir/test_file_24 became visible from 1 after 0s 2024-03-08T08:30:38.985 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:38.985 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.033 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.039 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.039 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.039 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_25", '"'"'r'"'"') as f: 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.040 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.041 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.082 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_25 2024-03-08T08:30:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_25 2024-03-08T08:30:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.098 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628302 Links: 1 2024-03-08T08:30:39.098 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.099 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.099 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.332205060 +0000 2024-03-08T08:30:39.099 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.333136719 +0000 2024-03-08T08:30:39.099 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.333136719 +0000 2024-03-08T08:30:39.099 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.099 INFO:tasks.cephfs.mount:File test_dir/test_file_25 became visible from 1 after 0s 2024-03-08T08:30:39.099 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.099 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.146 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.153 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_26", '"'"'r'"'"') as f: 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.154 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.155 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.156 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.158 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_26 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_26 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628303 Links: 1 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.335546759 +0000 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.336481885 +0000 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.336481885 +0000 2024-03-08T08:30:39.174 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.175 INFO:tasks.cephfs.mount:File test_dir/test_file_26 became visible from 1 after 0s 2024-03-08T08:30:39.175 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.175 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.221 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_27", '"'"'r'"'"') as f: 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.229 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.230 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.271 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_27 2024-03-08T08:30:39.275 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_27 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628304 Links: 1 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.339024912 +0000 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.339948160 +0000 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.339948160 +0000 2024-03-08T08:30:39.287 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.287 INFO:tasks.cephfs.mount:File test_dir/test_file_27 became visible from 1 after 0s 2024-03-08T08:30:39.288 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.288 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.334 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.342 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_28", '"'"'r'"'"') as f: 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.343 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.385 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_28 2024-03-08T08:30:39.393 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.400 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_28 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628305 Links: 1 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.342503085 +0000 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.343435924 +0000 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.343435924 +0000 2024-03-08T08:30:39.401 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.401 INFO:tasks.cephfs.mount:File test_dir/test_file_28 became visible from 1 after 0s 2024-03-08T08:30:39.401 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.401 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.448 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.457 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_29", '"'"'r'"'"') as f: 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.458 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.500 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_29 2024-03-08T08:30:39.511 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.515 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_29 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628306 Links: 1 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.345960505 +0000 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.346896140 +0000 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.346896140 +0000 2024-03-08T08:30:39.516 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.516 INFO:tasks.cephfs.mount:File test_dir/test_file_29 became visible from 1 after 0s 2024-03-08T08:30:39.516 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.516 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.563 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_30", '"'"'r'"'"') as f: 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.572 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.573 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.573 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.573 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.573 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.614 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_30 2024-03-08T08:30:39.628 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_30 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628307 Links: 1 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.349354028 +0000 2024-03-08T08:30:39.630 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.350282723 +0000 2024-03-08T08:30:39.631 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.350282723 +0000 2024-03-08T08:30:39.631 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.631 INFO:tasks.cephfs.mount:File test_dir/test_file_30 became visible from 1 after 0s 2024-03-08T08:30:39.631 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.631 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.677 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_31", '"'"'r'"'"') as f: 2024-03-08T08:30:39.686 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.687 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.728 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_31 2024-03-08T08:30:39.743 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_31 2024-03-08T08:30:39.743 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628308 Links: 1 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.352926238 +0000 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.353856195 +0000 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.353856195 +0000 2024-03-08T08:30:39.744 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.744 INFO:tasks.cephfs.mount:File test_dir/test_file_31 became visible from 1 after 0s 2024-03-08T08:30:39.744 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.744 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.745 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_32", '"'"'r'"'"') as f: 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.760 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.761 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.761 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.761 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.761 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.761 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.792 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.794 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_32 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_32 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628309 Links: 1 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.356470467 +0000 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.357419374 +0000 2024-03-08T08:30:39.810 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.357419374 +0000 2024-03-08T08:30:39.811 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.811 INFO:tasks.cephfs.mount:File test_dir/test_file_32 became visible from 1 after 0s 2024-03-08T08:30:39.811 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.811 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.857 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.858 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.865 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_33", '"'"'r'"'"') as f: 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.866 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.867 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.867 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.867 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.867 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:39.908 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_33 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_33 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628310 Links: 1 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:39.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.359841095 +0000 2024-03-08T08:30:39.924 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.360840130 +0000 2024-03-08T08:30:39.924 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.360840130 +0000 2024-03-08T08:30:39.924 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:39.924 INFO:tasks.cephfs.mount:File test_dir/test_file_33 became visible from 1 after 0s 2024-03-08T08:30:39.924 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:39.924 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:39.970 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.971 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_34", '"'"'r'"'"') as f: 2024-03-08T08:30:39.979 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:39.980 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.021 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_34 2024-03-08T08:30:40.036 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_34 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628311 Links: 1 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.363582322 +0000 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.364617963 +0000 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.364617963 +0000 2024-03-08T08:30:40.037 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.037 INFO:tasks.cephfs.mount:File test_dir/test_file_34 became visible from 1 after 0s 2024-03-08T08:30:40.037 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.037 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.083 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.086 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.092 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.092 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.092 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.092 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_35", '"'"'r'"'"') as f: 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.093 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.135 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_35 2024-03-08T08:30:40.151 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_35 2024-03-08T08:30:40.151 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.151 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628312 Links: 1 2024-03-08T08:30:40.151 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.152 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.152 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.367236626 +0000 2024-03-08T08:30:40.152 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.368265510 +0000 2024-03-08T08:30:40.152 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.368265510 +0000 2024-03-08T08:30:40.152 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.152 INFO:tasks.cephfs.mount:File test_dir/test_file_35 became visible from 1 after 0s 2024-03-08T08:30:40.152 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.152 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.195 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.199 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_36", '"'"'r'"'"') as f: 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.207 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.208 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.249 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_36 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_36 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628313 Links: 1 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.370821480 +0000 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.371866056 +0000 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.371866056 +0000 2024-03-08T08:30:40.265 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.265 INFO:tasks.cephfs.mount:File test_dir/test_file_36 became visible from 1 after 0s 2024-03-08T08:30:40.265 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.266 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.309 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.313 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_37", '"'"'r'"'"') as f: 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.321 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.322 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.322 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.322 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.334 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.336 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_37 2024-03-08T08:30:40.351 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_37 2024-03-08T08:30:40.351 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.351 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628314 Links: 1 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.374349216 +0000 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.375347247 +0000 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.375347247 +0000 2024-03-08T08:30:40.352 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.352 INFO:tasks.cephfs.mount:File test_dir/test_file_37 became visible from 1 after 0s 2024-03-08T08:30:40.352 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.352 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.398 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.399 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_38", '"'"'r'"'"') as f: 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.407 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.408 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.408 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.408 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.408 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.449 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_38 2024-03-08T08:30:40.453 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_38 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628315 Links: 1 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.377976923 +0000 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.378980004 +0000 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.378980004 +0000 2024-03-08T08:30:40.465 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.465 INFO:tasks.cephfs.mount:File test_dir/test_file_38 became visible from 1 after 0s 2024-03-08T08:30:40.465 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.465 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.513 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.513 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_39", '"'"'r'"'"') as f: 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.520 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.521 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.521 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.521 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.521 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.562 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_39 2024-03-08T08:30:40.570 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.578 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_39 2024-03-08T08:30:40.578 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628316 Links: 1 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.381499675 +0000 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.382448398 +0000 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.382448398 +0000 2024-03-08T08:30:40.579 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.579 INFO:tasks.cephfs.mount:File test_dir/test_file_39 became visible from 1 after 0s 2024-03-08T08:30:40.579 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.579 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.626 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.628 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.633 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_40", '"'"'r'"'"') as f: 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.634 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.635 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.635 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.635 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.635 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.676 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_40 2024-03-08T08:30:40.687 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.691 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_40 2024-03-08T08:30:40.691 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.691 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628317 Links: 1 2024-03-08T08:30:40.691 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.691 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.692 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.385040183 +0000 2024-03-08T08:30:40.692 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.386018828 +0000 2024-03-08T08:30:40.692 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.386018828 +0000 2024-03-08T08:30:40.692 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.692 INFO:tasks.cephfs.mount:File test_dir/test_file_40 became visible from 1 after 0s 2024-03-08T08:30:40.692 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.692 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.740 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.741 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_41", '"'"'r'"'"') as f: 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.747 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.748 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.789 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_41 2024-03-08T08:30:40.804 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_41 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628318 Links: 1 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.388820901 +0000 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.389807962 +0000 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.389807962 +0000 2024-03-08T08:30:40.805 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.805 INFO:tasks.cephfs.mount:File test_dir/test_file_41 became visible from 1 after 0s 2024-03-08T08:30:40.805 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.805 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.806 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.820 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_42", '"'"'r'"'"') as f: 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.821 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.853 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.853 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.855 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_42 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_42 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628319 Links: 1 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.392317104 +0000 2024-03-08T08:30:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.393302377 +0000 2024-03-08T08:30:40.872 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.393302377 +0000 2024-03-08T08:30:40.872 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.872 INFO:tasks.cephfs.mount:File test_dir/test_file_42 became visible from 1 after 0s 2024-03-08T08:30:40.872 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.872 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:40.919 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.922 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.927 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_43", '"'"'r'"'"') as f: 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:40.928 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:40.929 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:40.929 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:40.929 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:40.968 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:40.970 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_43 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_43 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628320 Links: 1 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.396171405 +0000 2024-03-08T08:30:40.986 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.397142735 +0000 2024-03-08T08:30:40.987 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.397142735 +0000 2024-03-08T08:30:40.987 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:40.987 INFO:tasks.cephfs.mount:File test_dir/test_file_43 became visible from 1 after 0s 2024-03-08T08:30:40.987 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:40.987 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.034 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.034 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_44", '"'"'r'"'"') as f: 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.042 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.043 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.043 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.084 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_44 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_44 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628321 Links: 1 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.400020993 +0000 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.401253495 +0000 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.401253495 +0000 2024-03-08T08:30:41.101 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.101 INFO:tasks.cephfs.mount:File test_dir/test_file_44 became visible from 1 after 0s 2024-03-08T08:30:41.101 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.101 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.148 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.149 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_45", '"'"'r'"'"') as f: 2024-03-08T08:30:41.157 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.158 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.160 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_45 2024-03-08T08:30:41.176 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_45 2024-03-08T08:30:41.176 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628322 Links: 1 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.403956565 +0000 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.405204855 +0000 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.405204855 +0000 2024-03-08T08:30:41.177 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.177 INFO:tasks.cephfs.mount:File test_dir/test_file_45 became visible from 1 after 0s 2024-03-08T08:30:41.177 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.177 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.222 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.224 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_46", '"'"'r'"'"') as f: 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.232 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.233 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.233 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.233 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.274 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_46 2024-03-08T08:30:41.277 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.289 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_46 2024-03-08T08:30:41.289 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628323 Links: 1 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.408010865 +0000 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.409262241 +0000 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.409262241 +0000 2024-03-08T08:30:41.290 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.290 INFO:tasks.cephfs.mount:File test_dir/test_file_46 became visible from 1 after 0s 2024-03-08T08:30:41.290 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.290 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.335 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.337 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_47", '"'"'r'"'"') as f: 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.345 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.346 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.346 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.346 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.346 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.387 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_47 2024-03-08T08:30:41.394 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.403 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_47 2024-03-08T08:30:41.403 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.403 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628324 Links: 1 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.411990983 +0000 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.413222201 +0000 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.413222201 +0000 2024-03-08T08:30:41.404 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.404 INFO:tasks.cephfs.mount:File test_dir/test_file_47 became visible from 1 after 0s 2024-03-08T08:30:41.404 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.404 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.449 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.451 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_48", '"'"'r'"'"') as f: 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.459 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.460 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.501 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_48 2024-03-08T08:30:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.517 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_48 2024-03-08T08:30:41.517 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628325 Links: 1 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.415922822 +0000 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.417001134 +0000 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.417001134 +0000 2024-03-08T08:30:41.518 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.518 INFO:tasks.cephfs.mount:File test_dir/test_file_48 became visible from 1 after 0s 2024-03-08T08:30:41.518 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.518 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.564 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.565 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_49", '"'"'r'"'"') as f: 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.573 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.574 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.574 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.574 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.574 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.615 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_49 2024-03-08T08:30:41.630 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.631 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_49 2024-03-08T08:30:41.631 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.631 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628326 Links: 1 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.419629988 +0000 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.420707311 +0000 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.420707311 +0000 2024-03-08T08:30:41.632 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.632 INFO:tasks.cephfs.mount:File test_dir/test_file_49 became visible from 1 after 0s 2024-03-08T08:30:41.632 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.632 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.678 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.679 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_50", '"'"'r'"'"') as f: 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.688 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.689 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.730 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_50 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_50 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628327 Links: 1 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.423672825 +0000 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.424706235 +0000 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.424706235 +0000 2024-03-08T08:30:41.746 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.746 INFO:tasks.cephfs.mount:File test_dir/test_file_50 became visible from 1 after 0s 2024-03-08T08:30:41.746 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.747 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.747 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.761 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_51", '"'"'r'"'"') as f: 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.762 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.795 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.795 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.796 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_51 2024-03-08T08:30:41.812 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_51 2024-03-08T08:30:41.812 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.812 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628328 Links: 1 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.427429658 +0000 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.428749390 +0000 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.428749390 +0000 2024-03-08T08:30:41.813 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.813 INFO:tasks.cephfs.mount:File test_dir/test_file_51 became visible from 1 after 0s 2024-03-08T08:30:41.813 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.813 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.859 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.860 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.861 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_52", '"'"'r'"'"') as f: 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.868 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.869 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.869 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.869 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.869 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.869 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:41.910 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_52 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_52 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628329 Links: 1 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:41.926 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.431475486 +0000 2024-03-08T08:30:41.929 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.432776541 +0000 2024-03-08T08:30:41.929 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.432776541 +0000 2024-03-08T08:30:41.929 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:41.929 INFO:tasks.cephfs.mount:File test_dir/test_file_52 became visible from 1 after 0s 2024-03-08T08:30:41.930 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:41.930 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:41.972 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.973 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.974 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_53", '"'"'r'"'"') as f: 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:41.981 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:41.982 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:41.982 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:41.982 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:41.982 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.023 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_53 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_53 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628330 Links: 1 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.435660699 +0000 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.437003722 +0000 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.437003722 +0000 2024-03-08T08:30:42.039 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.039 INFO:tasks.cephfs.mount:File test_dir/test_file_53 became visible from 1 after 0s 2024-03-08T08:30:42.040 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.040 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.085 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.087 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.088 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.095 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_54", '"'"'r'"'"') as f: 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.096 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.138 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_54 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_54 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628331 Links: 1 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.439653737 +0000 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.440677763 +0000 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.440677763 +0000 2024-03-08T08:30:42.154 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.154 INFO:tasks.cephfs.mount:File test_dir/test_file_54 became visible from 1 after 0s 2024-03-08T08:30:42.155 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.155 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.197 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.200 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.201 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.208 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_55", '"'"'r'"'"') as f: 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.209 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.251 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_55 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_55 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628332 Links: 1 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.443139872 +0000 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.444126394 +0000 2024-03-08T08:30:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.444126394 +0000 2024-03-08T08:30:42.268 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.268 INFO:tasks.cephfs.mount:File test_dir/test_file_55 became visible from 1 after 0s 2024-03-08T08:30:42.268 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.268 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.311 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.314 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.315 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_56", '"'"'r'"'"') as f: 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.324 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.325 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.337 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.339 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_56 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_56 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628333 Links: 1 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.446595314 +0000 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.447568909 +0000 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.447568909 +0000 2024-03-08T08:30:42.355 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.355 INFO:tasks.cephfs.mount:File test_dir/test_file_56 became visible from 1 after 0s 2024-03-08T08:30:42.356 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.356 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.400 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.401 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.402 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.410 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_57", '"'"'r'"'"') as f: 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.411 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.453 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_57 2024-03-08T08:30:42.455 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_57 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628334 Links: 1 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.450263901 +0000 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.451254252 +0000 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.451254252 +0000 2024-03-08T08:30:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.469 INFO:tasks.cephfs.mount:File test_dir/test_file_57 became visible from 1 after 0s 2024-03-08T08:30:42.469 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.470 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.514 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.515 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.516 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_58", '"'"'r'"'"') as f: 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.524 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.525 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.566 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_58 2024-03-08T08:30:42.572 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_58 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628335 Links: 1 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.454171496 +0000 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.455124522 +0000 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.455124522 +0000 2024-03-08T08:30:42.582 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.582 INFO:tasks.cephfs.mount:File test_dir/test_file_58 became visible from 1 after 0s 2024-03-08T08:30:42.582 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.583 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.628 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.630 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.630 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_59", '"'"'r'"'"') as f: 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.637 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.638 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.638 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.680 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_59 2024-03-08T08:30:42.689 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_59 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628336 Links: 1 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.457635185 +0000 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.458600307 +0000 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.458600307 +0000 2024-03-08T08:30:42.696 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.696 INFO:tasks.cephfs.mount:File test_dir/test_file_59 became visible from 1 after 0s 2024-03-08T08:30:42.696 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.697 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.741 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.746 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.746 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_60", '"'"'r'"'"') as f: 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.751 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.752 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.793 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_60 2024-03-08T08:30:42.808 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.809 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_60 2024-03-08T08:30:42.809 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.809 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628337 Links: 1 2024-03-08T08:30:42.809 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.809 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.461283926 +0000 2024-03-08T08:30:42.810 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.462350028 +0000 2024-03-08T08:30:42.810 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.462350028 +0000 2024-03-08T08:30:42.810 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.810 INFO:tasks.cephfs.mount:File test_dir/test_file_60 became visible from 1 after 0s 2024-03-08T08:30:42.810 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.810 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.854 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.855 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.857 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_61", '"'"'r'"'"') as f: 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.865 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.866 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.866 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.866 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.907 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_61 2024-03-08T08:30:42.920 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_61 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628338 Links: 1 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.464973971 +0000 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.465980338 +0000 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.465980338 +0000 2024-03-08T08:30:42.923 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.923 INFO:tasks.cephfs.mount:File test_dir/test_file_61 became visible from 1 after 0s 2024-03-08T08:30:42.923 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.923 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:42.924 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.938 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:42.938 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:42.938 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:42.938 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:42.938 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_62", '"'"'r'"'"') as f: 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:42.939 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:42.971 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.971 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:42.973 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_62 2024-03-08T08:30:42.989 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_62 2024-03-08T08:30:42.989 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628339 Links: 1 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.468828927 +0000 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.469896856 +0000 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.469896856 +0000 2024-03-08T08:30:42.990 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:42.990 INFO:tasks.cephfs.mount:File test_dir/test_file_62 became visible from 1 after 0s 2024-03-08T08:30:42.990 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:42.990 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.035 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.036 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.037 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.043 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_63", '"'"'r'"'"') as f: 2024-03-08T08:30:43.045 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.046 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.088 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_63 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_63 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628340 Links: 1 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.472531442 +0000 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.473560056 +0000 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.473560056 +0000 2024-03-08T08:30:43.104 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.104 INFO:tasks.cephfs.mount:File test_dir/test_file_63 became visible from 1 after 0s 2024-03-08T08:30:43.105 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.105 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.150 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.151 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.154 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_64", '"'"'r'"'"') as f: 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.159 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.160 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.160 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.160 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.162 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_64 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_64 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628341 Links: 1 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.476114196 +0000 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.477081106 +0000 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.477081106 +0000 2024-03-08T08:30:43.178 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.178 INFO:tasks.cephfs.mount:File test_dir/test_file_64 became visible from 1 after 0s 2024-03-08T08:30:43.178 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.179 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.224 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.225 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.226 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_65", '"'"'r'"'"') as f: 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.233 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.234 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.234 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.234 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.275 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_65 2024-03-08T08:30:43.279 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_65 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628342 Links: 1 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.479959091 +0000 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.480902234 +0000 2024-03-08T08:30:43.291 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.480902234 +0000 2024-03-08T08:30:43.292 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.292 INFO:tasks.cephfs.mount:File test_dir/test_file_65 became visible from 1 after 0s 2024-03-08T08:30:43.292 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.292 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.337 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.339 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.339 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_66", '"'"'r'"'"') as f: 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.347 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.348 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.348 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.348 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.348 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.389 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_66 2024-03-08T08:30:43.396 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_66 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628343 Links: 1 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.483423589 +0000 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.484474972 +0000 2024-03-08T08:30:43.405 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.484474972 +0000 2024-03-08T08:30:43.406 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.406 INFO:tasks.cephfs.mount:File test_dir/test_file_66 became visible from 1 after 0s 2024-03-08T08:30:43.406 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.406 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.451 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.452 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.453 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_67", '"'"'r'"'"') as f: 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.461 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.462 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.462 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.462 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.462 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.503 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_67 2024-03-08T08:30:43.515 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.519 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_67 2024-03-08T08:30:43.519 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.519 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628344 Links: 1 2024-03-08T08:30:43.519 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.520 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.520 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.487021588 +0000 2024-03-08T08:30:43.520 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.488105182 +0000 2024-03-08T08:30:43.520 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.488105182 +0000 2024-03-08T08:30:43.520 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.520 INFO:tasks.cephfs.mount:File test_dir/test_file_67 became visible from 1 after 0s 2024-03-08T08:30:43.520 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.520 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_68", '"'"'r'"'"') as f: 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.576 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.577 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.577 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.618 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_68 2024-03-08T08:30:43.632 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_68 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628345 Links: 1 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.490893287 +0000 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.491926239 +0000 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.491926239 +0000 2024-03-08T08:30:43.634 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.634 INFO:tasks.cephfs.mount:File test_dir/test_file_68 became visible from 1 after 0s 2024-03-08T08:30:43.634 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.635 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.682 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.682 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.683 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_69", '"'"'r'"'"') as f: 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.689 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.690 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.690 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.690 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.731 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_69 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_69 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628346 Links: 1 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.494501199 +0000 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.495516967 +0000 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.495516967 +0000 2024-03-08T08:30:43.747 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.747 INFO:tasks.cephfs.mount:File test_dir/test_file_69 became visible from 1 after 0s 2024-03-08T08:30:43.747 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.747 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.748 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_70", '"'"'r'"'"') as f: 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.763 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.764 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.764 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.764 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.764 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.796 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.797 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.797 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.798 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_70 2024-03-08T08:30:43.813 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_70 2024-03-08T08:30:43.813 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.813 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628347 Links: 1 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.498195136 +0000 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.499226999 +0000 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.499226999 +0000 2024-03-08T08:30:43.814 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.814 INFO:tasks.cephfs.mount:File test_dir/test_file_70 became visible from 1 after 0s 2024-03-08T08:30:43.814 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.814 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.861 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.862 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.862 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.862 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.868 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.868 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.868 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_71", '"'"'r'"'"') as f: 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.869 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:43.911 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_71 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_71 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628348 Links: 1 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.501745590 +0000 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.502798497 +0000 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.502798497 +0000 2024-03-08T08:30:43.927 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:43.927 INFO:tasks.cephfs.mount:File test_dir/test_file_71 became visible from 1 after 0s 2024-03-08T08:30:43.928 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:43.928 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:43.982 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_72", '"'"'r'"'"') as f: 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:43.983 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:43.984 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:43.984 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.025 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_72 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_72 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628349 Links: 1 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.505343409 +0000 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.506349399 +0000 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.506349399 +0000 2024-03-08T08:30:44.041 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.041 INFO:tasks.cephfs.mount:File test_dir/test_file_72 became visible from 1 after 0s 2024-03-08T08:30:44.042 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.042 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.087 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.087 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.088 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.089 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_73", '"'"'r'"'"') as f: 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.097 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.138 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_73 2024-03-08T08:30:44.154 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_73 2024-03-08T08:30:44.154 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.154 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628350 Links: 1 2024-03-08T08:30:44.154 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.154 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.155 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.509218799 +0000 2024-03-08T08:30:44.155 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.510581816 +0000 2024-03-08T08:30:44.155 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.510581816 +0000 2024-03-08T08:30:44.155 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.155 INFO:tasks.cephfs.mount:File test_dir/test_file_73 became visible from 1 after 0s 2024-03-08T08:30:44.155 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.155 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.198 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.202 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.202 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.203 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.209 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.209 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.209 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.209 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_74", '"'"'r'"'"') as f: 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.210 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.252 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_74 2024-03-08T08:30:44.267 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_74 2024-03-08T08:30:44.267 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.267 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628351 Links: 1 2024-03-08T08:30:44.267 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.268 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.268 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.512982187 +0000 2024-03-08T08:30:44.268 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.514044142 +0000 2024-03-08T08:30:44.268 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.514044142 +0000 2024-03-08T08:30:44.268 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.268 INFO:tasks.cephfs.mount:File test_dir/test_file_74 became visible from 1 after 0s 2024-03-08T08:30:44.268 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.268 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.313 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.315 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.316 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.316 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_75", '"'"'r'"'"') as f: 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.324 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.325 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.338 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.340 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_75 2024-03-08T08:30:44.356 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_75 2024-03-08T08:30:44.356 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.356 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628352 Links: 1 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.516474555 +0000 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.517547856 +0000 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.517547856 +0000 2024-03-08T08:30:44.357 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.357 INFO:tasks.cephfs.mount:File test_dir/test_file_75 became visible from 1 after 0s 2024-03-08T08:30:44.357 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.357 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.402 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.403 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.404 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.404 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_76", '"'"'r'"'"') as f: 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.412 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.413 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.454 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_76 2024-03-08T08:30:44.457 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_76 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628353 Links: 1 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.520103948 +0000 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.521140071 +0000 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.521140071 +0000 2024-03-08T08:30:44.470 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.470 INFO:tasks.cephfs.mount:File test_dir/test_file_76 became visible from 1 after 0s 2024-03-08T08:30:44.471 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.471 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.516 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.517 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.517 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.518 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_77", '"'"'r'"'"') as f: 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.525 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.526 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.526 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.526 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.526 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.526 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.567 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_77 2024-03-08T08:30:44.574 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.582 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_77 2024-03-08T08:30:44.582 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.582 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628354 Links: 1 2024-03-08T08:30:44.582 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.582 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.583 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.523636712 +0000 2024-03-08T08:30:44.583 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.524675950 +0000 2024-03-08T08:30:44.583 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.524675950 +0000 2024-03-08T08:30:44.583 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.583 INFO:tasks.cephfs.mount:File test_dir/test_file_77 became visible from 1 after 0s 2024-03-08T08:30:44.583 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.583 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.629 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.631 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.631 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.632 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_78", '"'"'r'"'"') as f: 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.638 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.639 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:30:44.662 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:30:44.680 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_78 2024-03-08T08:30:44.691 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.696 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_78 2024-03-08T08:30:44.696 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.696 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628355 Links: 1 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.527323779 +0000 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.528366423 +0000 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.528366423 +0000 2024-03-08T08:30:44.697 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.697 INFO:tasks.cephfs.mount:File test_dir/test_file_78 became visible from 1 after 0s 2024-03-08T08:30:44.697 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.697 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.743 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.744 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.745 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.745 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_79", '"'"'r'"'"') as f: 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.757 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.758 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.758 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.758 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.799 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_79 2024-03-08T08:30:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_79 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628356 Links: 1 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.530963357 +0000 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.532045604 +0000 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.532045604 +0000 2024-03-08T08:30:44.815 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.815 INFO:tasks.cephfs.mount:File test_dir/test_file_79 became visible from 1 after 0s 2024-03-08T08:30:44.815 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.816 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.857 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.857 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.858 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.863 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_80", '"'"'r'"'"') as f: 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.870 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.871 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.871 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.871 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.871 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:44.912 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_80 2024-03-08T08:30:44.922 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.926 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_80 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628357 Links: 1 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.534584962 +0000 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.535648612 +0000 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.535648612 +0000 2024-03-08T08:30:44.928 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:44.928 INFO:tasks.cephfs.mount:File test_dir/test_file_80 became visible from 1 after 0s 2024-03-08T08:30:44.928 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:44.929 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:44.972 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.972 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_81", '"'"'r'"'"') as f: 2024-03-08T08:30:44.983 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:44.984 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.025 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_81 2024-03-08T08:30:45.037 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.038 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.038 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_81 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628358 Links: 1 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.538272318 +0000 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.539313865 +0000 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.539313865 +0000 2024-03-08T08:30:45.041 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.041 INFO:tasks.cephfs.mount:File test_dir/test_file_81 became visible from 1 after 0s 2024-03-08T08:30:45.042 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.042 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.045 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_82", '"'"'r'"'"') as f: 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.060 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.061 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.061 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.089 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.091 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_82 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_82 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628359 Links: 1 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.541881151 +0000 2024-03-08T08:30:45.107 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.542914414 +0000 2024-03-08T08:30:45.108 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.542914414 +0000 2024-03-08T08:30:45.108 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.108 INFO:tasks.cephfs.mount:File test_dir/test_file_82 became visible from 1 after 0s 2024-03-08T08:30:45.108 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.108 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.152 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.152 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.153 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.155 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_83", '"'"'r'"'"') as f: 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.163 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.164 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.164 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.205 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_83 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_83 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628360 Links: 1 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.545292556 +0000 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.546302550 +0000 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.546302550 +0000 2024-03-08T08:30:45.221 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.221 INFO:tasks.cephfs.mount:File test_dir/test_file_83 became visible from 1 after 0s 2024-03-08T08:30:45.222 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.222 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.226 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.240 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.240 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_84", '"'"'r'"'"') as f: 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.241 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.269 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.270 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_84 2024-03-08T08:30:45.281 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_84 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628361 Links: 1 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.286 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.548794329 +0000 2024-03-08T08:30:45.287 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.549802408 +0000 2024-03-08T08:30:45.287 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.549802408 +0000 2024-03-08T08:30:45.287 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.287 INFO:tasks.cephfs.mount:File test_dir/test_file_84 became visible from 1 after 0s 2024-03-08T08:30:45.287 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.287 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.334 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.339 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_85", '"'"'r'"'"') as f: 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.342 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.343 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.384 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_85 2024-03-08T08:30:45.398 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.400 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_85 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628362 Links: 1 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.552428910 +0000 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.553482804 +0000 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.553482804 +0000 2024-03-08T08:30:45.401 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.401 INFO:tasks.cephfs.mount:File test_dir/test_file_85 became visible from 1 after 0s 2024-03-08T08:30:45.401 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.401 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.448 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.453 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.454 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_86", '"'"'r'"'"') as f: 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.456 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.457 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.457 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.457 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.457 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.457 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.498 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_86 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_86 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628363 Links: 1 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.556035150 +0000 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.557025275 +0000 2024-03-08T08:30:45.514 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.557025275 +0000 2024-03-08T08:30:45.515 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.515 INFO:tasks.cephfs.mount:File test_dir/test_file_86 became visible from 1 after 0s 2024-03-08T08:30:45.515 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.515 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.517 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_87", '"'"'r'"'"') as f: 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.532 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.533 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.533 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.562 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.564 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_87 2024-03-08T08:30:45.568 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.569 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.569 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.579 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_87 2024-03-08T08:30:45.579 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628364 Links: 1 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.559544357 +0000 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.560567762 +0000 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.560567762 +0000 2024-03-08T08:30:45.580 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.580 INFO:tasks.cephfs.mount:File test_dir/test_file_87 became visible from 1 after 0s 2024-03-08T08:30:45.580 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.580 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.628 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.635 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_88", '"'"'r'"'"') as f: 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.636 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.678 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_88 2024-03-08T08:30:45.681 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.682 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.683 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.693 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_88 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628365 Links: 1 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.563677524 +0000 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.564719324 +0000 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.564719324 +0000 2024-03-08T08:30:45.694 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.694 INFO:tasks.cephfs.mount:File test_dir/test_file_88 became visible from 1 after 0s 2024-03-08T08:30:45.694 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.695 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.742 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_89", '"'"'r'"'"') as f: 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.749 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.750 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.750 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.750 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.750 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.752 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_89 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_89 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628366 Links: 1 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.567253270 +0000 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.568341548 +0000 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.568341548 +0000 2024-03-08T08:30:45.768 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.768 INFO:tasks.cephfs.mount:File test_dir/test_file_89 became visible from 1 after 0s 2024-03-08T08:30:45.768 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.769 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.797 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.799 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.800 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_90", '"'"'r'"'"') as f: 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.812 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.813 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.813 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.816 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.817 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_90 2024-03-08T08:30:45.833 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_90 2024-03-08T08:30:45.833 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628367 Links: 1 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.570964153 +0000 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.571975786 +0000 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.571975786 +0000 2024-03-08T08:30:45.834 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.834 INFO:tasks.cephfs.mount:File test_dir/test_file_90 became visible from 1 after 0s 2024-03-08T08:30:45.834 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.834 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.863 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.864 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.864 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.864 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_91", '"'"'r'"'"') as f: 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.879 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.880 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.880 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.880 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.882 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.884 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_91 2024-03-08T08:30:45.899 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_91 2024-03-08T08:30:45.899 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.899 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628368 Links: 1 2024-03-08T08:30:45.899 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.900 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.900 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.574465088 +0000 2024-03-08T08:30:45.900 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.575565111 +0000 2024-03-08T08:30:45.900 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.575565111 +0000 2024-03-08T08:30:45.900 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.900 INFO:tasks.cephfs.mount:File test_dir/test_file_91 became visible from 1 after 0s 2024-03-08T08:30:45.900 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.900 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:45.948 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_92", '"'"'r'"'"') as f: 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:45.955 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:45.956 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:45.956 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:45.956 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:45.956 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:45.976 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.977 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.978 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_92 2024-03-08T08:30:45.978 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_92 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628369 Links: 1 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.578255460 +0000 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.579246949 +0000 2024-03-08T08:30:45.994 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.579246949 +0000 2024-03-08T08:30:45.995 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:45.995 INFO:tasks.cephfs.mount:File test_dir/test_file_92 became visible from 1 after 0s 2024-03-08T08:30:45.995 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:45.995 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.041 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_93", '"'"'r'"'"') as f: 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.050 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.051 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.051 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.051 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.051 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.051 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.090 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.090 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.090 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.092 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_93 2024-03-08T08:30:46.092 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_93 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628370 Links: 1 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.581695250 +0000 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.582855491 +0000 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.582855491 +0000 2024-03-08T08:30:46.108 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.109 INFO:tasks.cephfs.mount:File test_dir/test_file_93 became visible from 1 after 0s 2024-03-08T08:30:46.109 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.109 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.156 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.162 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.162 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_94", '"'"'r'"'"') as f: 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.163 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.164 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.200 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.202 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_94 2024-03-08T08:30:46.204 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.205 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.205 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_94 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628371 Links: 1 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.585461003 +0000 2024-03-08T08:30:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.586531165 +0000 2024-03-08T08:30:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.586531165 +0000 2024-03-08T08:30:46.219 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.219 INFO:tasks.cephfs.mount:File test_dir/test_file_94 became visible from 1 after 0s 2024-03-08T08:30:46.219 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.219 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.266 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_95", '"'"'r'"'"') as f: 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.274 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.275 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.316 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_95 2024-03-08T08:30:46.316 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.317 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.317 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.319 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.332 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_95 2024-03-08T08:30:46.332 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.332 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628372 Links: 1 2024-03-08T08:30:46.332 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.333 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.333 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.589280821 +0000 2024-03-08T08:30:46.333 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.590279051 +0000 2024-03-08T08:30:46.333 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.590279051 +0000 2024-03-08T08:30:46.333 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.333 INFO:tasks.cephfs.mount:File test_dir/test_file_95 became visible from 1 after 0s 2024-03-08T08:30:46.333 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.333 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.340 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.355 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_96", '"'"'r'"'"') as f: 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.356 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.357 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.357 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.380 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.382 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_96 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_96 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628373 Links: 1 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.592938657 +0000 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.593911745 +0000 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.593911745 +0000 2024-03-08T08:30:46.398 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.399 INFO:tasks.cephfs.mount:File test_dir/test_file_96 became visible from 1 after 0s 2024-03-08T08:30:46.399 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.399 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.404 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.406 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.407 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.407 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_97", '"'"'r'"'"') as f: 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.419 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.420 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.447 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.448 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_97 2024-03-08T08:30:46.459 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.464 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_97 2024-03-08T08:30:46.464 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628374 Links: 1 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.596491876 +0000 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.597485432 +0000 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.597485432 +0000 2024-03-08T08:30:46.465 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.465 INFO:tasks.cephfs.mount:File test_dir/test_file_97 became visible from 1 after 0s 2024-03-08T08:30:46.465 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.465 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.512 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.518 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.519 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.519 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_98", '"'"'r'"'"') as f: 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.521 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.522 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.522 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.522 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.522 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.522 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.563 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_98 2024-03-08T08:30:46.576 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_98 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628375 Links: 1 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.599973996 +0000 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.601036675 +0000 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.601036675 +0000 2024-03-08T08:30:46.579 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.579 INFO:tasks.cephfs.mount:File test_dir/test_file_98 became visible from 1 after 0s 2024-03-08T08:30:46.580 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:30:46.580 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:30:46.627 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.631 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.632 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> stdin-killer -- python3 -c ' 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> import fcntl 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> import sys 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> import time 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.634 DEBUG:teuthology.orchestra.run.smithi156:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> with open("/home/ubuntu/cephtest/mnt.1/test_dir/test_file_99", '"'"'r'"'"') as f: 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> print("open_background: keeping file open", file=sys.stderr) 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> try: 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> if os.read(0, 4096) == b"": 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> break 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> except BlockingIOError: 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> pass 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(2) 2024-03-08T08:30:46.635 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-08T08:30:46.635 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.637 DEBUG:teuthology.orchestra.run.smithi156:> stat /home/ubuntu/cephtest/mnt.1/test_dir/test_file_99 2024-03-08T08:30:46.653 INFO:teuthology.orchestra.run.smithi156.stdout: File: /home/ubuntu/cephtest/mnt.1/test_dir/test_file_99 2024-03-08T08:30:46.653 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 2 Blocks: 1 IO Block: 4194304 regular file 2024-03-08T08:30:46.653 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 29h/41d Inode: 1099511628376 Links: 1 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-08 08:30:32.603861911 +0000 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-08 08:30:32.604970474 +0000 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-08 08:30:32.604970474 +0000 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-03-08T08:30:46.654 INFO:tasks.cephfs.mount:File test_dir/test_file_99 became visible from 1 after 0s 2024-03-08T08:30:46.654 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T08:30:46.654 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 counter dump 2024-03-08T08:30:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.702 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.745 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.747 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.747 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.747 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.768+0000 7f64301d7640 1 Processor -- start 2024-03-08T08:30:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 -- start start 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f642815baa0 0x7f642806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 0x7f64280cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642815be80 con 0x7f642806abd0 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642806f3b0 con 0x7f642815baa0 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f64301d7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642806f690 con 0x7f64280cd380 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f642df73640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f642815baa0 0x7f642806bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f642df73640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f642815baa0 0x7f642806bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45382/0 (socket says 172.21.15.12:45382) 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f642df73640 1 -- 172.21.15.12:0/4259711055 learned_addr learned my addr 172.21.15.12:0/4259711055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.769+0000 7f642e774640 1 --1- 172.21.15.12:0/4259711055 >> v1:172.21.15.12:6789/0 conn(0x7f642806abd0 0x7f642806b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46368/0 (socket says 172.21.15.12:46368) 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642d772640 1 --2- 172.21.15.12:0/4259711055 >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 0x7f64280cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642df73640 1 -- 172.21.15.12:0/4259711055 >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 msgr2=0x7f64280cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642df73640 1 --2- 172.21.15.12:0/4259711055 >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 0x7f64280cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642df73640 1 -- 172.21.15.12:0/4259711055 >> v1:172.21.15.12:6789/0 conn(0x7f642806abd0 legacy=0x7f642806b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642df73640 1 -- 172.21.15.12:0/4259711055 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642806f970 con 0x7f642815baa0 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642cf71640 1 -- 172.21.15.12:0/4259711055 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777491909 0 0) 0x7f642815be80 con 0x7f642806abd0 2024-03-08T08:30:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.770+0000 7f642d772640 1 --2- 172.21.15.12:0/4259711055 >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 0x7f64280cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:30:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f642df73640 1 --2- 172.21.15.12:0/4259711055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f642806bf10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f642806ea60 tx=0x7f642402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caec1dfb7bb106b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f642cf71640 1 -- 172.21.15.12:0/4259711055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424034070 con 0x7f642815baa0 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f642cf71640 1 -- 172.21.15.12:0/4259711055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424030860 con 0x7f642815baa0 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f642cf71640 1 -- 172.21.15.12:0/4259711055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424030c30 con 0x7f642815baa0 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 -- 172.21.15.12:0/4259711055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 msgr2=0x7f642806bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 --2- 172.21.15.12:0/4259711055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f642806bf10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f642806ea60 tx=0x7f642402b9a0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 -- 172.21.15.12:0/4259711055 shutdown_connections 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 --2- 172.21.15.12:0/4259711055 >> v2:172.21.15.156:3300/0 conn(0x7f64280cd380 0x7f64280cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 --2- 172.21.15.12:0/4259711055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f642806bf10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 -- 172.21.15.12:0/4259711055 >> 172.21.15.12:0/4259711055 conn(0x7f642806db00 msgr2=0x7f642806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 -- 172.21.15.12:0/4259711055 shutdown_connections 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.771+0000 7f64301d7640 1 -- 172.21.15.12:0/4259711055 wait complete. 2024-03-08T08:30:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 Processor -- start 2024-03-08T08:30:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 -- start start 2024-03-08T08:30:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f642806abd0 0x7f642814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64280cd380 0x7f642814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642806f040 con 0x7f642815baa0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642806ff30 con 0x7f64280cd380 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f64301d7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64280af8b0 con 0x7f642806abd0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f642e774640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f642e774640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45398/0 (socket says 172.21.15.12:45398) 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.773+0000 7f642e774640 1 -- 172.21.15.12:0/1845464468 learned_addr learned my addr 172.21.15.12:0/1845464468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642df73640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f642806abd0 0x7f642814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642d772640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64280cd380 0x7f642814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 -- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f642806abd0 msgr2=0x7f642814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f642806abd0 0x7f642814a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 -- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64280cd380 msgr2=0x7f642814d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64280cd380 0x7f642814d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 -- 172.21.15.12:0/1845464468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642814d780 con 0x7f642815baa0 2024-03-08T08:30:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f642e774640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f642000c7d0 tx=0x7f642000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f641e7fc640 1 -- 172.21.15.12:0/1845464468 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420018070 con 0x7f642815baa0 2024-03-08T08:30:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f641e7fc640 1 -- 172.21.15.12:0/1845464468 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642000a5e0 con 0x7f642815baa0 2024-03-08T08:30:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64280b3f40 con 0x7f642815baa0 2024-03-08T08:30:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f641e7fc640 1 -- 172.21.15.12:0/1845464468 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642000aa10 con 0x7f642815baa0 2024-03-08T08:30:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.774+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64280b4500 con 0x7f642815baa0 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.776+0000 7f641e7fc640 1 -- 172.21.15.12:0/1845464468 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f642001b070 con 0x7f642815baa0 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.776+0000 7f641e7fc640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 0x7f640c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.776+0000 7f641e7fc640 1 -- 172.21.15.12:0/1845464468 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6420089680 con 0x7f642815baa0 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.776+0000 7f642df73640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 0x7f640c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.777+0000 7f642df73640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 0x7f640c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6424036000 tx=0x7f642403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.777+0000 7f6432dbb740 1 Processor -- start 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.777+0000 7f6432dbb740 1 -- start start 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb350ed8a0 0x55fb350edc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb3511dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb35057e30 con 0x55fb350f18f0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb35000d20 con 0x55fb3511d630 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb35053ed0 con 0x55fb350ed8a0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642d772640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642d772640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45410/0 (socket says 172.21.15.12:45410) 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642d772640 1 -- 172.21.15.12:0/1137344824 learned_addr learned my addr 172.21.15.12:0/1137344824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642df73640 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb350ed8a0 0x55fb350edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642e774640 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb3511dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.778+0000 7f642d772640 1 -- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb350ed8a0 msgr2=0x55fb350edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f642d772640 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb350ed8a0 0x55fb350edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f642d772640 1 -- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 msgr2=0x55fb3511dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f642d772640 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb3511dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f642d772640 1 -- 172.21.15.12:0/1137344824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb34d6c3f0 con 0x55fb350f18f0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f642d772640 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f641800abc0 tx=0x7f641800f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95531d14b21b86b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f640b7fe640 1 -- 172.21.15.12:0/1137344824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800fe80 con 0x55fb350f18f0 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f640b7fe640 1 -- 172.21.15.12:0/1137344824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6418013440 con 0x55fb350f18f0 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.779+0000 7f640b7fe640 1 -- 172.21.15.12:0/1137344824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64180137f0 con 0x55fb350f18f0 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- 172.21.15.12:0/1137344824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 msgr2=0x55fb350f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f641800abc0 tx=0x7f641800f240 comp rx=0 tx=0).stop 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- 172.21.15.12:0/1137344824 shutdown_connections 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb3511dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb350f18f0 0x55fb350f3390 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1137344824 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb350ed8a0 0x55fb350edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- 172.21.15.12:0/1137344824 >> 172.21.15.12:0/1137344824 conn(0x55fb350ef920 msgr2=0x55fb350f11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- 172.21.15.12:0/1137344824 shutdown_connections 2024-03-08T08:30:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- 172.21.15.12:0/1137344824 wait complete. 2024-03-08T08:30:46.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 -- start start 2024-03-08T08:30:46.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.780+0000 7f6432dbb740 1 Processor -- start 2024-03-08T08:30:46.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb35112760 0x55fb350f2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb3511f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb350f1d80 con 0x55fb3511d630 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb34ff6390 con 0x55fb35112760 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb350f2060 con 0x55fb35110eb0 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f6432dbb740 10 client.0 _ll_register_callbacks cb 0x55fb35013c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f642df73640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f642d772640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb35112760 0x55fb350f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f642df73640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45426/0 (socket says 172.21.15.12:45426) 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f642df73640 1 -- 172.21.15.12:0/877823666 learned_addr learned my addr 172.21.15.12:0/877823666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.781+0000 7f642e774640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb3511f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 -- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 msgr2=0x55fb3511f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb3511f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 -- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb35112760 msgr2=0x55fb350f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb35112760 0x55fb350f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 -- 172.21.15.12:0/877823666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb34e367d0 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f642df73640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6424009930 tx=0x7f642402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d931650d0f004c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424054060 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6424039650 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6418013440 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f642405a050 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6432dbb740 10 client.5529 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55fb35116670 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6418013440 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.782+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f642405d020 con 0x55fb3511d630 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 10 client.5529 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 10 client.5529 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:46.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 20 client.5529 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb3511f970 0x55fb3511fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 4 client.5529 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb35065370 con 0x55fb3511f970 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f642d772640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb3511f970 0x55fb3511fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f642d772640 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb3511f970 0x55fb3511fd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26377ed17224964 server_cookie=f31785774e1c8688 in_seq=0 out_seq=0 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6408ff9640 10 client.5529 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6408ff9640 1 -- 172.21.15.12:0/877823666 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb35065370 con 0x55fb3511f970 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.783+0000 7f6408ff9640 10 client.5529 handle_command_reply: tid=0 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 client.5529 shutdown 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 msgr2=0x55fb350f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6424009930 tx=0x7f642402e000 comp rx=0 tx=0).stop 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 shutdown_connections 2024-03-08T08:30:46.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb3511f970 0x55fb3511fd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb3511f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb35112760 0x55fb350f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6408ff9640 0 client.5529 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 --2- 172.21.15.12:0/877823666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb3511d630 0x55fb350f2240 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 >> 172.21.15.12:0/877823666 conn(0x55fb350ed8a0 msgr2=0x55fb350ec5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 shutdown_connections 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 1 -- 172.21.15.12:0/877823666 wait complete. 2024-03-08T08:30:46.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.784+0000 7f6432dbb740 20 client.5529 trim_cache size 0 max 16384 2024-03-08T08:30:46.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 Processor -- start 2024-03-08T08:30:46.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 -- start start 2024-03-08T08:30:46.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb350f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35147b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb35153500 con 0x55fb35112760 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64281ffc90 con 0x55fb3511d630 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.796+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64281feec0 con 0x55fb35110eb0 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642d772640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642d772640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45434/0 (socket says 172.21.15.12:45434) 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642d772640 1 -- 172.21.15.12:0/906886672 learned_addr learned my addr 172.21.15.12:0/906886672 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb350f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642e774640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 -- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 msgr2=0x55fb35147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35147b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 -- 172.21.15.12:0/906886672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 msgr2=0x55fb350f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 -- 172.21.15.12:0/906886672 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb350e9630 con 0x55fb35110eb0 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642d772640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f642df73640 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb350f21b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6424009960 tx=0x7f6424009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a2b9b21c131da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f640affd640 1 -- 172.21.15.12:0/906886672 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424054070 con 0x55fb35110eb0 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f640affd640 1 -- 172.21.15.12:0/906886672 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424039450 con 0x55fb35110eb0 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.797+0000 7f640affd640 1 -- 172.21.15.12:0/906886672 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424034040 con 0x55fb35110eb0 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 msgr2=0x55fb350f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb350f21b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6424009960 tx=0x7f6424009240 comp rx=0 tx=0).stop 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- 172.21.15.12:0/906886672 shutdown_connections 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35112760 0x55fb350f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 --2- 172.21.15.12:0/906886672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35110eb0 0x55fb350f21b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- 172.21.15.12:0/906886672 >> 172.21.15.12:0/906886672 conn(0x55fb350ed8a0 msgr2=0x55fb35121200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- 172.21.15.12:0/906886672 shutdown_connections 2024-03-08T08:30:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- 172.21.15.12:0/906886672 wait complete. 2024-03-08T08:30:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 -- start start 2024-03-08T08:30:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.798+0000 7f6432dbb740 1 Processor -- start 2024-03-08T08:30:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35112760 0x55fb3511c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35138010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64280caf40 con 0x55fb35110eb0 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64280a90d0 con 0x55fb3511d630 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb3510f990 con 0x55fb35112760 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f6432dbb740 10 client.0 _ll_register_callbacks cb 0x55fb35113670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f642df73640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f642df73640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45436/0 (socket says 172.21.15.12:45436) 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f642df73640 1 -- 172.21.15.12:0/1483379953 learned_addr learned my addr 172.21.15.12:0/1483379953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f642e774640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.799+0000 7f642d772640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35112760 0x55fb3511c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 -- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35112760 msgr2=0x55fb3511c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35112760 0x55fb3511c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 -- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 msgr2=0x55fb35138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35138010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 -- 172.21.15.12:0/1483379953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb35138be0 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642e774640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35138010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f642df73640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6424004650 tx=0x7f6424004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22be1ce753eb524 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424063060 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6424039b70 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6424039450 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6424069050 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 10 client.5535 fetch_fsmap learned FSMap version 15 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55fb35138f80 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424039450 con 0x55fb35110eb0 2024-03-08T08:30:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f642406c020 con 0x55fb35110eb0 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 10 client.5535 fetch_fsmap finished waiting for FSMap version 15 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 10 client.5535 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 20 client.5535 populate_metadata read hostname 'smithi012' 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb35117e20 0x55fb35138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 4 client.5535 mds_command: new command op to 5277 tid=0 multi_id=0 [{"prefix": "counter dump"}] 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.800+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 --> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] -- command(tid 0: {"prefix": "counter dump"}) v1 -- 0x55fb3505f6c0 con 0x55fb35117e20 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f642d772640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb35117e20 0x55fb35138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6408ff9640 10 client.5535 ms_handle_connect on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f642d772640 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb35117e20 0x55fb35138550 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd83b5e7ae2b1ca6 server_cookie=84ce0ee4d5318f51 in_seq=0 out_seq=0 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6408ff9640 1 -- 172.21.15.12:0/1483379953 <== mds.0 v2:172.21.15.12:6836/2820296658 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26955 (crc 0 0 0) 0x55fb3505f6c0 con 0x55fb35117e20 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6408ff9640 10 client.5535 handle_command_reply: tid=0 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6432dbb740 1 client.5535 shutdown 2024-03-08T08:30:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 msgr2=0x55fb351a3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.801+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6424004650 tx=0x7f6424004730 comp rx=0 tx=0).stop 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 shutdown_connections 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:6836/2820296658,v1:172.21.15.12:6837/2820296658] conn(0x55fb35117e20 0x55fb35138550 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb3511d630 0x55fb35138010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55fb35112760 0x55fb3511c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 --2- 172.21.15.12:0/1483379953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb35110eb0 0x55fb351a3e00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6408ff9640 0 client.5535 ms_handle_reset on v2:172.21.15.12:6836/2820296658 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 >> 172.21.15.12:0/1483379953 conn(0x55fb350ed8a0 msgr2=0x55fb35117060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 shutdown_connections 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 1 -- 172.21.15.12:0/1483379953 wait complete. 2024-03-08T08:30:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f6432dbb740 20 client.5535 trim_cache size 0 max 16384 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker": [ 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "0" 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "1" 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "2" 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.811 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_ready_timeouts": 0, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_connection_idle_timeouts": 0 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": [ 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 4103, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 4085, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 1293510, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 1784321, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 9, 2024-03-08T08:30:46.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.223956169, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.108658957, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.185300636, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.001390658, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 4083, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.083386448, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000020422 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 8055, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.001243114, 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000154 2024-03-08T08:30:46.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 113520, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 14736 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": [ 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 181, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 166, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 56940, 2024-03-08T08:30:46.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 1894064, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 7, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 4, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.027806136, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.009169195, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.019887919, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.002268832, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 165, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.183721253, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001113462 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 283, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000053435, 2024-03-08T08:30:46.815 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000188 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 100064 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": [ 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1055, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 1057, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 259968, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 4696752, 2024-03-08T08:30:46.816 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 12, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 8, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.066932244, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.023817648, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.019555158, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.019107364, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1055, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.157294100, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000149093 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 6, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000626, 2024-03-08T08:30:46.817 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000104 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": [ 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 1, 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-08T08:30:46.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-MDSRank": [ 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2086, 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.595080315, 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000285273 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.819 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-PurgeQueue": [ 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000018932, 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000009466 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.820 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": [ 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "request": 1117, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "reply": 1117, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "reply_latency": { 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1117, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.590647667, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000528780 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_reply": 0, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "forward": 0, 2024-03-08T08:30:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_complete": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_keys": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_commit": 12, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_split": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_merge": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 1011, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_top": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_bottom": 1011, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pin_tail": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pinned": 1011, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_expired": 0, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_with_caps": 1002, 2024-03-08T08:30:46.822 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": 1104, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "subtrees": 2, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse": 1234, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_hit": 1232, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_forward": 0, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_discover": 0, 2024-03-08T08:30:46.825 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_dir_fetch": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_remote_ino": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_lock": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "load_cent": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "q": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "exported": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "exported_inodes": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "imported": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "imported_inodes": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_dir_fetch": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_backtrace_fetch": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_peer_discover": 0, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rfiles": 1000, 2024-03-08T08:30:46.826 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rbytes": 2890, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rsnaps": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_set_tag": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_inotable_repaired": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_inodes": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_file_inodes": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_inode_file_caps": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_revoke": 25, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_grant": 1000, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T08:30:46.827 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps": 2014, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps_dirty": 1000, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_cap_release": 0, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "process_request_cap_release": 1002 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_cache": [ 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update": 0, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update_receipt": 0, 2024-03-08T08:30:46.828 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_try_discover": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_send_discover": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_handle_discover": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_delayed": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_enqueuing": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_created": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_enqueued": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_reintegrated": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_migrated": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_processing": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_enqueued": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_prioritized": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_started": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_completed": 0, 2024-03-08T08:30:46.829 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_exportdir": 0, 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_flush": 0, 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragmentdir": 0, 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragstats": 0, 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_inodestats": 0 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics": [ 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-08T08:30:46.830 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "5277" 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "num_clients": 2 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_client_metrics-cephfs": [ 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.4796", 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 1856, 2024-03-08T08:30:46.831 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 14, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 243, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 243, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 81, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 83, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 83, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 83, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 0, 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 0 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": { 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "client": "client.5466", 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": "0" 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_hits": 10010, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_miss": 3, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_read_latency": 0.000000000, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_write_latency": 0.000000000, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_metadata_latency": 0.000000000, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_hits": 1001, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_lease_miss": 0, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_files": 0, 2024-03-08T08:30:46.833 INFO:teuthology.orchestra.run.smithi012.stdout: "opened_inodes": 1002, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "pinned_icaps": 1002, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "total_inodes": 1002, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_ops": 0, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "total_read_size": 0, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_ops": 1000, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "total_write_size": 2890 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_log": [ 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "evlrg": 0, 2024-03-08T08:30:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: "evadd": 3010, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "evex": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "evtrm": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "ev": 3010, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "evexg": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "evexd": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segadd": 5, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segex": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segtrm": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "seg": 5, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segmjr": 2, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segexg": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "segexd": 0, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "expos": 4194304, 2024-03-08T08:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout: "wrpos": 10064971, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "rdpos": 0, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "jlat": { 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1020, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 1.375286395, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001348319 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "replayed": 0 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_mem": [ 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: "ino": 1014, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "ino+": 1014, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "ino-": 0, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dir": 13, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dir+": 13, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dir-": 0, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dn": 1011, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dn+": 1011, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "dn-": 0, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "cap": 1104, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "cap+": 1104, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "cap-": 0, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "rss": 61504, 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: "heap": 215392 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.837 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_server": [ 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_client_request": 1143, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_server_request": 0, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_request": 1117, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_session": 1008, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_peer_request": 0, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "req_create_latency": { 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1000, 2024-03-08T08:30:46.838 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.503472675, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000503472 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getattr_latency": { 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 14, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.033490001, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.002392142 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getfilelock_latency": { 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "req_link_latency": { 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.839 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookup_latency": { 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 101, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.050576186, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000500754 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookuphash_latency": { 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupino_latency": { 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.840 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupname_latency": { 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupparent_latency": { 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupsnap_latency": { 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.841 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lssnap_latency": { 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mkdir_latency": { 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.002857981, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.002857981 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mknod_latency": { 2024-03-08T08:30:46.842 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mksnap_latency": { 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "req_open_latency": { 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "req_readdir_latency": { 2024-03-08T08:30:46.843 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rename_latency": { 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "req_renamesnap_latency": { 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "req_snapdiff_latency": { 2024-03-08T08:30:46.844 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmdir_latency": { 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmsnap_latency": { 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmxattr_latency": { 2024-03-08T08:30:46.845 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setattr_latency": { 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000250824, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000250824 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setdirlayout_latency": { 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setfilelock_latency": { 2024-03-08T08:30:46.846 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setlayout_latency": { 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setxattr_latency": { 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "req_symlink_latency": { 2024-03-08T08:30:46.847 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "req_unlink_latency": { 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_revoke_eviction": 0, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_acquisition_throttle": 0, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getvxattr_latency": { 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.848 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_sessions": [ 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "session_count": 2, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "session_add": 2, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "session_remove": 0, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_open": 2, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_stale": 0, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "total_load": 974, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "average_load": 487, 2024-03-08T08:30:46.849 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_session_uptime": 24, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "mdthresh_evicted": 0 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": [ 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-08T08:30:46.850 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T08:30:46.851 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 4827795, 2024-03-08T08:30:46.852 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 1233, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 5312, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 75, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-08T08:30:46.853 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 4046104, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 23756, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "objecter": [ 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "op_active": 0, 2024-03-08T08:30:46.854 INFO:teuthology.orchestra.run.smithi012.stdout: "op_laggy": 0, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send": 1070, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send_bytes": 5964453, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_resend": 37, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_reply": 1070, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_latency": { 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1070, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 2.095981604, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.001958861 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.855 INFO:teuthology.orchestra.run.smithi012.stdout: "op_inflight": 0, 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "oplen_avg": { 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1070, 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 1105 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "op": 1070, 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "op_r": 0, 2024-03-08T08:30:46.856 INFO:teuthology.orchestra.run.smithi012.stdout: "op_w": 1070, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "op_rmw": 0, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "op_pg": 0, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_stat": 0, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_create": 11, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_read": 0, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_write": 1020, 2024-03-08T08:30:46.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writefull": 8, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writesame": 0, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_append": 0, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_zero": 0, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_truncate": 0, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_delete": 7, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_mapext": 0, 2024-03-08T08:30:46.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_sparse_read": 0, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_clonerange": 0, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_getxattr": 0, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_setxattr": 22, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_cmpxattr": 0, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_rmxattr": 0, 2024-03-08T08:30:46.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_resetxattrs": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_call": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_watch": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_notify": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls_filter": 0, 2024-03-08T08:30:46.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_other": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_active": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_send": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_resend": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_ping": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_active": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_send": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_resend": 0, 2024-03-08T08:30:46.861 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_active": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_send": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_resend": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_active": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_send": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_resend": 0, 2024-03-08T08:30:46.862 INFO:teuthology.orchestra.run.smithi012.stdout: "command_active": 0, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "command_send": 0, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "command_resend": 0, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "map_epoch": 35, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "map_full": 0, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "map_inc": 1, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_sessions": 6, 2024-03-08T08:30:46.863 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_open": 6, 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_close": 0, 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_laggy": 0, 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_wr": 30, 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_rd": 0, 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_del": 7 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: "oft": [ 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_objs": 1, 2024-03-08T08:30:46.865 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_kv_pairs": 0, 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_updates": 0, 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_removes": 0 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: "osd-slow-ops": [ 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_ops_count": 0 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.867 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "purge_queue": [ 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops": 0, 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T08:30:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing": 0, 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_high_water": 0, 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed_ops": 0, 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed": 0, 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_item_in_journal": 0 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.869 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": [ 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:46.870 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 5331, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 1125153, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 5331, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:46.871 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 5331, 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 1125153, 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.872 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_bytes": [ 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.873 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:46.874 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 1070, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 5964453, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1070, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 5964453, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.875 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_ops": [ 2024-03-08T08:30:46.876 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 1024, 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:46.877 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 1070, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 1070, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1070, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 1070, 2024-03-08T08:30:46.878 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.879 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle": [ 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:46.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:46.881 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 0, 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 0, 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle-0x559bb8a9a6e0": [ 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: "labels": {}, 2024-03-08T08:30:46.883 INFO:teuthology.orchestra.run.smithi012.stdout: "counters": { 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 3010, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 5870667, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-08T08:30:46.884 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 3010, 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 1020, 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 5870667, 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-08T08:30:46.885 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-08T08:30:46.886 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:30:46.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 msgr2=0x7f640c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 0x7f640c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6424036000 tx=0x7f642403b040 comp rx=0 tx=0).stop 2024-03-08T08:30:46.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 msgr2=0x7f64280a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:30:46.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f642000c7d0 tx=0x7f642000cc90 comp rx=0 tx=0).stop 2024-03-08T08:30:46.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 shutdown_connections 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f640c06b4d0 0x7f640c06d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642815baa0 0x7f64280a9550 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64280cd380 0x7f642814d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 --2- 172.21.15.12:0/1845464468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f642806abd0 0x7f642814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 >> 172.21.15.12:0/1845464468 conn(0x7f642806db00 msgr2=0x7f642806bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:30:46.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 shutdown_connections 2024-03-08T08:30:46.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:30:46.802+0000 7f64301d7640 1 -- 172.21.15.12:0/1845464468 wait complete. 2024-03-08T08:30:46.891 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:46.892 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.892 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.924 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.928 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.974 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.975 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:46.978 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.039 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.040 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.040 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.047 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.091 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.154 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.157 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.164 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.229 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.230 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.271 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.336 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.400 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.450 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.455 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.457 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.457 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.520 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.564 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.571 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.571 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.571 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.630 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.637 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.684 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.744 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.753 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.798 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.801 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.801 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.818 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.865 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.866 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.866 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.866 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.884 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.892 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:47.950 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:47.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.044 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.093 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.158 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.203 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.206 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.207 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.207 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.268 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.317 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.318 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.319 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.320 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.342 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.382 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.406 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.408 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.408 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.408 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.449 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.461 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.515 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.520 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.522 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.522 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.522 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.579 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.628 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.633 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.634 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.635 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.695 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.703 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.747 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.748 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.749 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.749 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.814 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.861 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.861 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.862 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.868 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.893 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:48.927 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.930 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.977 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.977 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:48.980 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.040 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.042 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.042 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.050 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.093 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.157 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.157 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.158 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.160 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.167 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.232 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.232 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.273 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.286 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.338 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.344 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.345 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.345 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.402 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.452 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.458 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.458 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.460 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.522 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.573 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.573 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.573 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.632 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.639 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.686 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.687 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.687 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.746 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.800 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.803 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.803 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.820 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.867 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.868 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.868 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.869 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.894 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:49.952 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.980 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.980 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.981 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:49.981 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.046 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.094 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.094 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.094 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.095 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.160 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.205 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.270 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.320 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.320 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.322 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.323 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.345 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.384 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.407 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.410 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.410 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.410 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.451 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.464 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.516 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.523 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.524 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.524 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.524 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.581 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.631 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.638 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.639 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.749 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.749 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.751 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.817 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.863 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.863 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.864 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.870 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.895 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:50.929 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.933 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:50.982 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.042 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.044 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.044 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.050 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.095 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.159 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.159 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.160 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.160 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.169 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.233 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.275 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.288 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.346 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.346 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.404 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.454 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.460 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.461 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.462 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.523 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.568 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.575 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.575 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.575 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.634 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.641 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.748 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.757 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.802 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.805 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.805 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.822 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.869 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.870 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.870 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.871 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.896 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:51.955 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.980 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:51.984 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.048 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.096 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.096 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.096 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.096 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.162 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.207 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.210 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.211 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.211 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.272 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.321 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.322 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.324 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.325 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.347 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.386 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.409 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.411 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.412 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.413 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.453 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.466 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.519 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.524 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.525 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.525 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.526 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.583 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.633 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.637 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.638 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.640 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.641 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.700 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.706 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.751 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.751 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.753 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.753 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.819 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.865 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.865 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.867 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.897 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:52.931 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.935 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.981 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.981 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:52.984 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.044 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.046 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.046 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.052 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.097 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.161 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.161 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.162 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.162 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.171 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.235 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.236 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.236 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.276 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.290 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.349 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.407 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.456 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.461 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.463 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.464 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.525 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.570 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.577 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.577 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.577 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.636 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.643 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.690 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.690 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.690 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.750 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.762 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.804 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.807 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.807 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.824 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.871 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.873 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.898 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:53.956 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.983 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:53.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.050 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.098 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.098 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.098 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.098 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.209 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.212 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.213 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.213 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.275 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.323 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.324 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.326 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.327 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.349 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.389 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.412 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.412 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.414 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.414 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.455 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.468 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.521 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.527 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.527 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.527 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.528 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.635 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.639 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.640 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.640 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.643 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.702 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.753 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.754 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.755 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.755 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.821 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.867 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.867 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.869 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.874 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.899 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:54.933 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.937 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.983 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.983 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:54.986 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.046 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.048 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.048 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.054 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.100 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.172 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.237 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.237 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.278 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.292 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.345 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.350 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.350 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.351 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.409 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.458 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.462 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.465 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.466 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.528 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.572 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.579 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.579 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.579 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.638 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.645 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.692 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.692 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.693 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.752 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.806 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.809 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.809 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.827 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.873 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.874 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.875 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.900 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:55.959 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.987 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.987 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:55.988 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.052 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.098 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.099 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.100 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.100 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.167 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.209 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.214 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.215 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.215 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.325 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.327 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.328 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.329 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.351 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.391 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.414 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.414 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.415 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.416 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.457 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.470 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.523 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.529 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.529 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.529 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.530 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.587 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.637 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.642 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.642 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.642 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.645 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.704 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.711 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.755 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.756 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.758 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.758 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.822 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.869 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.869 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.871 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.876 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.901 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:56.935 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.939 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.985 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:56.988 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.048 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.050 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.050 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.056 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.165 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.165 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.165 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.166 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.174 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.239 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.239 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.240 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.281 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.294 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.347 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.352 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.354 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.411 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.461 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.464 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.467 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.468 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.530 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.574 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.581 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.582 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.582 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.694 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.695 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.695 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.752 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.763 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.811 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.811 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.829 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.874 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.875 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.876 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.895 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.902 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:57.961 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.987 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.989 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.989 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:57.990 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.054 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.100 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.102 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.102 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.102 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.168 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.211 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.216 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.217 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.217 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.279 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.328 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.329 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.330 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.331 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.459 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.470 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.525 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.531 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.531 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.531 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.533 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.589 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.639 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.644 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.644 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.644 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.647 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.704 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.713 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.758 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.758 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.825 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.871 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.872 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.878 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.903 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:58.937 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.941 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.987 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.988 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:58.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.049 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.052 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.053 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.058 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.167 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.167 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.168 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.168 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.177 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.241 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.242 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.242 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.286 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.296 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.349 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.354 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.355 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.356 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.413 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.463 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.466 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.469 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.469 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.532 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.643 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.649 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.696 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.697 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.697 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.755 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.763 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.810 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.813 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.814 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.903 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:30:59.963 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.990 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:30:59.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.056 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.103 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.104 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.104 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.104 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.171 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.214 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.219 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.219 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.219 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.281 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.330 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.331 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.332 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.333 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.355 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.395 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.418 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.418 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.418 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.419 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.462 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.472 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.527 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.533 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.533 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.533 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.535 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.591 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.641 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.646 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.646 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.647 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.649 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.704 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.715 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.761 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.826 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.873 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.874 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.874 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.880 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.904 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:00.939 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.943 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.989 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.989 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:00.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.051 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.054 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.055 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.061 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.106 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.169 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.169 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.170 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.170 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.179 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.243 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.243 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.244 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.285 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.298 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.351 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.356 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.357 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.358 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.415 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.465 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.469 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.471 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.472 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.534 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.579 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.584 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.584 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.586 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.645 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.651 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.698 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.699 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.699 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.757 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.765 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.812 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.816 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.816 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.878 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.879 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.880 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.899 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.905 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:01.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.058 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.105 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.106 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.106 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.107 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.173 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.283 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.332 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.333 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.334 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.335 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.357 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.397 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.420 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.420 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.420 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.421 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.464 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.474 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.535 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.535 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.535 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.537 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.593 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.642 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.648 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.648 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.649 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.651 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.707 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.715 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.760 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.762 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.762 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.764 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.829 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.876 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.876 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.876 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.905 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:02.942 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.945 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.991 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:02.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.053 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.056 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.057 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.063 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.108 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.171 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.172 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.172 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.172 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.181 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.244 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.245 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.245 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.287 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.299 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.359 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.359 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.360 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.417 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.467 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.470 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.473 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.474 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.536 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.581 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.586 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.586 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.587 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.647 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.653 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.701 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.701 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.701 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.759 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.768 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.814 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.818 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.818 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.835 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.901 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.906 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:03.967 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:03.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.061 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.107 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.108 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.108 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.108 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.175 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.216 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.222 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.340 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.342 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.342 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.359 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.398 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.422 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.422 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.423 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.423 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.466 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.476 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.531 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.537 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.537 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.538 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.539 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.596 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.645 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.650 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.650 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.651 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.653 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.709 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.717 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.762 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.763 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.764 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.766 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.831 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.907 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:04.944 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.947 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.993 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.994 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:04.997 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.055 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.058 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.059 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.065 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.110 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.172 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.173 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.173 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.174 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.183 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.247 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.247 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.247 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.289 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.302 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.355 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.361 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.361 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.362 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.469 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.473 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.474 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.475 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.538 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.583 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.588 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.588 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.589 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.649 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.703 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.703 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.703 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.770 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.817 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.820 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.820 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.837 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.903 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.908 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:05.969 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.997 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.997 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:05.998 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.063 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.109 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.110 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.110 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.111 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.175 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.218 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.223 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.224 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.225 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.287 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.337 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.361 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.400 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.424 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.424 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.425 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.425 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.468 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.479 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.533 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.539 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.539 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.540 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.541 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.598 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.647 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.653 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.719 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.765 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.765 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.766 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.768 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.833 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.879 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.879 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.880 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.909 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:06.946 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.949 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.995 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.996 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:06.999 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.055 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.060 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.061 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.067 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.112 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.175 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.175 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.175 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.176 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.185 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.249 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.249 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.249 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.291 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.304 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.358 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.361 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.362 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.364 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.421 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.471 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.474 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.477 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.478 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.540 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.585 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.590 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.590 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.592 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.651 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.703 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.763 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.771 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.819 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.820 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.822 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.839 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.904 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.910 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:07.971 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.997 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.999 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:07.999 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.000 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.065 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.111 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.112 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.113 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.113 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.178 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.220 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.225 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.225 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.226 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.290 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.338 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.339 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.342 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.363 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.401 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.426 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.426 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.427 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.427 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.470 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.481 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.535 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.541 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.542 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.542 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.544 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.600 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.649 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.654 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.712 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.722 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.766 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.767 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.767 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.769 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.835 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.881 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.911 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:08.948 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.952 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.997 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:08.998 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.001 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.057 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.069 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.113 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.177 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.177 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.177 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.178 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.187 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.251 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.251 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.251 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.293 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.306 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.360 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.364 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.364 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.366 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.423 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.473 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.476 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.478 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.479 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.542 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.587 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.592 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.592 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.594 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.653 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.660 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.706 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.706 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.706 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.774 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.820 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.823 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.824 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.841 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.906 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.912 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:09.974 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:09.999 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.001 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.002 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.002 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.066 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.113 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.113 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.114 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.115 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.180 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.222 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.227 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.292 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.340 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.341 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.344 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.364 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.404 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.428 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.429 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.429 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.429 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.472 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.483 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.537 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.543 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.544 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.544 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.545 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.602 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.651 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.656 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.715 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.724 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.768 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.769 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.769 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.771 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.837 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.882 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.883 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.912 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:10.950 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.954 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:10.999 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.000 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.060 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.064 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.065 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.115 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.179 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.179 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.180 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.180 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.252 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.253 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.254 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.295 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.308 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.362 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.366 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.366 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.426 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.475 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.478 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.481 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.481 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.545 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.589 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.594 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.594 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.596 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.662 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.706 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.708 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.708 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.767 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.775 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.822 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.825 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.826 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.844 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.890 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.909 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:11.913 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:11.976 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.001 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.069 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.115 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.116 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.116 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.117 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.182 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.225 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.229 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.230 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.230 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.294 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.342 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.343 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.344 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.346 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.367 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.406 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.429 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.474 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.485 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.539 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.546 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.546 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.546 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.548 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.604 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.653 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.715 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.726 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.769 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.770 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.771 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.773 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.839 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.884 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.885 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.914 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:12.952 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:12.956 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.001 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.002 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.006 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.067 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.067 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.073 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.117 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.181 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.181 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.181 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.181 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.191 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.255 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.255 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.256 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.298 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.310 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.364 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.369 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.428 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.477 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.480 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.481 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.483 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.547 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.592 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.596 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.596 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.598 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.664 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.707 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.710 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.710 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.769 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.778 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.825 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.827 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.827 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.846 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.892 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.892 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.892 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.911 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:13.915 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:13.978 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.071 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.117 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.118 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.119 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.119 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.184 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.226 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.231 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.232 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.232 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.296 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.345 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.346 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.346 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.369 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.408 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.429 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.433 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.434 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.434 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.476 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.487 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.541 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.548 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.548 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.548 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.550 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.606 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.655 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.661 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.661 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.661 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.664 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.716 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.728 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:31:14.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:31:14.772 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.773 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.773 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.775 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.841 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.886 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.887 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.895 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.916 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:14.954 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:14.958 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.003 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.004 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.006 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.064 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.069 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.069 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.075 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.119 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.183 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.183 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.183 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.184 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.193 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.257 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.257 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.258 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.300 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.312 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.366 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.370 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.371 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.430 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.479 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.482 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.484 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.485 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.549 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.594 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.598 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.599 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.600 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.666 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.707 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.712 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.712 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.771 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.779 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.827 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.828 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.829 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.848 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.913 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:15.916 DEBUG:tasks.cephfs.test_admin:jos debug hits 2024-03-08T08:31:15.916 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-08T08:31:15.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-08T08:31:15.975 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-08T08:31:15.979 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.006 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.007 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.007 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.022 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-08T08:31:16.022 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:31:16.023 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:16.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:16.073 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.120 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.120 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.121 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.121 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.186 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.186+0000 7f0b2bd25640 1 Processor -- start 2024-03-08T08:31:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.186+0000 7f0b2bd25640 1 -- start start 2024-03-08T08:31:16.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b2bd25640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b2bd25640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 0x7f0b24161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b2bd25640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b24161fb0 con 0x7f0b240d0e10 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b2bd25640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b24161d80 con 0x7f0b2415df10 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b2bd25640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d1d60 con 0x7f0b240d0a30 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b292c0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 0x7f0b24161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:39048/0 (socket says 172.21.15.12:39048) 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 -- 172.21.15.12:0/928877840 learned_addr learned my addr 172.21.15.12:0/928877840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 -- 172.21.15.12:0/928877840 >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 msgr2=0x7f0b24161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 --2- 172.21.15.12:0/928877840 >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 0x7f0b24161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 -- 172.21.15.12:0/928877840 >> v1:172.21.15.156:6789/0 conn(0x7f0b240d0e10 legacy=0x7f0b240b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b29ac1640 1 -- 172.21.15.12:0/928877840 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24162290 con 0x7f0b240d0a30 2024-03-08T08:31:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.187+0000 7f0b292c0640 1 --2- 172.21.15.12:0/928877840 >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 0x7f0b24161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.188+0000 7f0b29ac1640 1 --2- 172.21.15.12:0/928877840 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009870 tx=0x7f0b2002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b727c4264b04d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.188+0000 7f0b28abf640 1 -- 172.21.15.12:0/928877840 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20034070 con 0x7f0b240d0a30 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.188+0000 7f0b28abf640 1 -- 172.21.15.12:0/928877840 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20030840 con 0x7f0b240d0a30 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.188+0000 7f0b28abf640 1 -- 172.21.15.12:0/928877840 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20030c10 con 0x7f0b240d0a30 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 -- 172.21.15.12:0/928877840 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 msgr2=0x7f0b240b95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/928877840 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009870 tx=0x7f0b2002b980 comp rx=0 tx=0).stop 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 -- 172.21.15.12:0/928877840 shutdown_connections 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/928877840 >> v2:172.21.15.156:3300/0 conn(0x7f0b2415df10 0x7f0b24161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/928877840 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b240d0a30 0x7f0b240b95a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 -- 172.21.15.12:0/928877840 >> 172.21.15.12:0/928877840 conn(0x7f0b2406c710 msgr2=0x7f0b2406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 -- 172.21.15.12:0/928877840 shutdown_connections 2024-03-08T08:31:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.189+0000 7f0b2bd25640 1 -- 172.21.15.12:0/928877840 wait complete. 2024-03-08T08:31:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.190+0000 7f0b2bd25640 1 Processor -- start 2024-03-08T08:31:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.190+0000 7f0b2bd25640 1 -- start start 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b240d0a30 0x7f0b241586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b2415df10 0x7f0b240b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d12a0 con 0x7f0b240d0e10 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d1fb0 con 0x7f0b240d0a30 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2bd25640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d34e0 con 0x7f0b2415df10 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50982/0 (socket says 172.21.15.12:50982) 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 -- 172.21.15.12:0/657582163 learned_addr learned my addr 172.21.15.12:0/657582163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b29ac1640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b240d0a30 0x7f0b241586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b2a2c2640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b2415df10 0x7f0b240b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 -- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b2415df10 msgr2=0x7f0b240b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b2415df10 0x7f0b240b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 -- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b240d0a30 msgr2=0x7f0b241586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b240d0a30 0x7f0b241586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.191+0000 7f0b292c0640 1 -- 172.21.15.12:0/657582163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b241594f0 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b292c0640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e900 tx=0x7f0b1400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14017070 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400ca00 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b240b5420 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400cdf0 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.192+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b240b3400 con 0x7f0b240d0e10 2024-03-08T08:31:16.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.193+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b240749c0 con 0x7f0b240d0e10 2024-03-08T08:31:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.193+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b1401a070 con 0x7f0b240d0e10 2024-03-08T08:31:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.194+0000 7f0b1a7fc640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 0x7f0b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.194+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0b14089d10 con 0x7f0b240d0e10 2024-03-08T08:31:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.194+0000 7f0b29ac1640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 0x7f0b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.194+0000 7f0b29ac1640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 0x7f0b0806d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0b20036000 tx=0x7f0b20008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.198+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0b140584c0 con 0x7f0b240d0e10 2024-03-08T08:31:16.227 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.298 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.347 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.347 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.350 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.371 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.378+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b240d1630 con 0x7f0b240d0e10 2024-03-08T08:31:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.379+0000 7f0b1a7fc640 1 -- 172.21.15.12:0/657582163 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16859 (secure 0 0 0) 0x7f0b1405c560 con 0x7f0b240d0e10 2024-03-08T08:31:16.381 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:16.381 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":37,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:55.107876+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.35","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 msgr2=0x7f0b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 0x7f0b0806d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0b20036000 tx=0x7f0b20008030 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 msgr2=0x7f0b240b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e900 tx=0x7f0b1400edc0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 shutdown_connections 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f0b0806b4d0 0x7f0b0806d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0b2415df10 0x7f0b240b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b240d0e10 0x7f0b240b48b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 --2- 172.21.15.12:0/657582163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b240d0a30 0x7f0b241586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 >> 172.21.15.12:0/657582163 conn(0x7f0b2406c710 msgr2=0x7f0b240c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:16.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 shutdown_connections 2024-03-08T08:31:16.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.381+0000 7f0b2bd25640 1 -- 172.21.15.12:0/657582163 wait complete. 2024-03-08T08:31:16.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:30:09.998602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:30:11.090210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:31:16.424 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:31:16.425 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.431 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.435 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.435 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.435 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.478 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.489 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.528+0000 7f1f1c615640 1 Processor -- start 2024-03-08T08:31:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.528+0000 7f1f1c615640 1 -- start start 2024-03-08T08:31:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1c615640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f140b6b30 0x7f1f140b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1c615640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1c615640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f140d1d90 con 0x7f1f14161260 2024-03-08T08:31:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1c615640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f140b9ac0 con 0x7f1f140b6b30 2024-03-08T08:31:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1c615640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f140d2070 con 0x7f1f1406c160 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f19bb0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f140b6b30 0x7f1f140b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1a3b1640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f14161260 0x7f1f140b93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46946/0 (socket says 172.21.15.12:46946) 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1a3b1640 1 -- 172.21.15.12:0/4042869140 learned_addr learned my addr 172.21.15.12:0/4042869140 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f1abb2640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f19bb0640 1 -- 172.21.15.12:0/4042869140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 msgr2=0x7f1f1406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f19bb0640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f19bb0640 1 -- 172.21.15.12:0/4042869140 >> v1:172.21.15.12:6789/0 conn(0x7f1f14161260 legacy=0x7f1f140b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.529+0000 7f1f19bb0640 1 -- 172.21.15.12:0/4042869140 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f140d2350 con 0x7f1f140b6b30 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f19bb0640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f140b6f90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1f040097a0 tx=0x7f1f0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed613c85603af84 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f193af640 1 -- 172.21.15.12:0/4042869140 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04035020 con 0x7f1f140b6b30 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f193af640 1 -- 172.21.15.12:0/4042869140 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f04031830 con 0x7f1f140b6b30 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f193af640 1 -- 172.21.15.12:0/4042869140 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04031c00 con 0x7f1f140b6b30 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 -- 172.21.15.12:0/4042869140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 msgr2=0x7f1f140b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f140b6f90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1f040097a0 tx=0x7f1f0402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 -- 172.21.15.12:0/4042869140 shutdown_connections 2024-03-08T08:31:16.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 --2- 172.21.15.12:0/4042869140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f140b6f90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.530+0000 7f1f1c615640 1 -- 172.21.15.12:0/4042869140 >> 172.21.15.12:0/4042869140 conn(0x7f1f14072930 msgr2=0x7f1f140d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:16.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.531+0000 7f1f1c615640 1 -- 172.21.15.12:0/4042869140 shutdown_connections 2024-03-08T08:31:16.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.531+0000 7f1f1c615640 1 -- 172.21.15.12:0/4042869140 wait complete. 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 Processor -- start 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 -- start start 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 0x7f1f1414de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f140d2e30 con 0x7f1f140b6b30 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f140b2ec0 con 0x7f1f14161260 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f1c615640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f140d2fb0 con 0x7f1f1406c160 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f19bb0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f19bb0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50998/0 (socket says 172.21.15.12:50998) 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.532+0000 7f1f19bb0640 1 -- 172.21.15.12:0/2311278221 learned_addr learned my addr 172.21.15.12:0/2311278221 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f1a3b1640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f1abb2640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 0x7f1f1414de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 -- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 msgr2=0x7f1f1415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1415a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 -- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 msgr2=0x7f1f1414de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 0x7f1f1414de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 -- 172.21.15.12:0/2311278221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14157d20 con 0x7f1f140b6b30 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f1abb2640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 0x7f1f1414de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f19bb0640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009890 tx=0x7f1f040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04041070 con 0x7f1f140b6b30 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0403b920 con 0x7f1f140b6b30 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f140c8b90 con 0x7f1f140b6b30 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04035040 con 0x7f1f140b6b30 2024-03-08T08:31:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.533+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f140c9150 con 0x7f1f140b6b30 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.534+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f140749c0 con 0x7f1f140b6b30 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.535+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f0404c060 con 0x7f1f140b6b30 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.535+0000 7f1f0affd640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 0x7f1ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.535+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1f040abc10 con 0x7f1f140b6b30 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.535+0000 7f1f1a3b1640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 0x7f1ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.535+0000 7f1f1a3b1640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 0x7f1ef806d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000f60 tx=0x7f1f10009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.539+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f0407aa20 con 0x7f1f140b6b30 2024-03-08T08:31:16.542 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.550 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.550 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.550 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.552 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.608 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.657 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.663 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.663 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.663 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.666 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.718+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f1414e3c0 con 0x7f1f140b6b30 2024-03-08T08:31:16.719 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.718+0000 7f1f0affd640 1 -- 172.21.15.12:0/2311278221 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f0407a320 con 0x7f1f140b6b30 2024-03-08T08:31:16.719 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 msgr2=0x7f1ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 0x7f1ef806d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000f60 tx=0x7f1f10009040 comp rx=0 tx=0).stop 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 msgr2=0x7f1f141577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009890 tx=0x7f1f040046c0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 shutdown_connections 2024-03-08T08:31:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f1ef806b4d0 0x7f1ef806d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f14161260 0x7f1f1414de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f140b6b30 0x7f1f141577e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 --2- 172.21.15.12:0/2311278221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f1406c160 0x7f1f1415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 >> 172.21.15.12:0/2311278221 conn(0x7f1f14072930 msgr2=0x7f1f140b8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 shutdown_connections 2024-03-08T08:31:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.720+0000 7f1f1c615640 1 -- 172.21.15.12:0/2311278221 wait complete. 2024-03-08T08:31:16.730 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:31:16.774 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.774 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.775 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.777 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.836+0000 7fc2a649d640 1 Processor -- start 2024-03-08T08:31:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.836+0000 7fc2a649d640 1 -- start start 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a649d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 0x7fc2a00cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a649d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2a006bef0 0x7fc2a00cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a649d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a00d0290 con 0x7fc2a0099b90 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a649d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00d0570 con 0x7fc2a006bef0 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a649d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00d0850 con 0x7fc2a015e0d0 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a4a3a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2a006bef0 0x7fc2a00cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a4a3a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2a006bef0 0x7fc2a00cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51000/0 (socket says 172.21.15.12:51000) 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/1285734734 learned_addr learned my addr 172.21.15.12:0/1285734734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.837+0000 7fc29f7fe640 1 --2- 172.21.15.12:0/1285734734 >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 0x7fc2a00cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/1285734734 >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 msgr2=0x7fc2a00cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc2a4a3a640 1 --2- 172.21.15.12:0/1285734734 >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 0x7fc2a00cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/1285734734 >> v1:172.21.15.156:6789/0 conn(0x7fc2a0099b90 legacy=0x7fc2a0099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/1285734734 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00d0b30 con 0x7fc2a006bef0 2024-03-08T08:31:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc29f7fe640 1 --2- 172.21.15.12:0/1285734734 >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 0x7fc2a00cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.838+0000 7fc2a4a3a640 1 --2- 172.21.15.12:0/1285734734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a006bef0 0x7fc2a00cbfc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0072a60 tx=0x7fc29802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56cedf5af4578fe server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc29effd640 1 -- 172.21.15.12:0/1285734734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29802cb20 con 0x7fc2a006bef0 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc29effd640 1 -- 172.21.15.12:0/1285734734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29802ccc0 con 0x7fc2a006bef0 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc29effd640 1 -- 172.21.15.12:0/1285734734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298034610 con 0x7fc2a006bef0 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 -- 172.21.15.12:0/1285734734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a006bef0 msgr2=0x7fc2a00cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 --2- 172.21.15.12:0/1285734734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a006bef0 0x7fc2a00cbfc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0072a60 tx=0x7fc29802bf80 comp rx=0 tx=0).stop 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 -- 172.21.15.12:0/1285734734 shutdown_connections 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 --2- 172.21.15.12:0/1285734734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a006bef0 0x7fc2a00cbfc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 --2- 172.21.15.12:0/1285734734 >> v2:172.21.15.156:3300/0 conn(0x7fc2a015e0d0 0x7fc2a00cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 -- 172.21.15.12:0/1285734734 >> 172.21.15.12:0/1285734734 conn(0x7fc2a00b3460 msgr2=0x7fc2a00b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 -- 172.21.15.12:0/1285734734 shutdown_connections 2024-03-08T08:31:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.839+0000 7fc2a649d640 1 -- 172.21.15.12:0/1285734734 wait complete. 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.840+0000 7fc2a649d640 1 Processor -- start 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.840+0000 7fc2a649d640 1 -- start start 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2a006bef0 0x7fc2a00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a0099b90 0x7fc2a0160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a009b310 con 0x7fc2a0158e30 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a009b190 con 0x7fc2a0099b90 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a649d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00d1330 con 0x7fc2a006bef0 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a4a3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a4a3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51012/0 (socket says 172.21.15.12:51012) 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/923857565 learned_addr learned my addr 172.21.15.12:0/923857565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc29f7fe640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a0099b90 0x7fc2a0160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.841+0000 7fc29ffff640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2a006bef0 0x7fc2a00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2a006bef0 msgr2=0x7fc2a00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2a006bef0 0x7fc2a00b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a0099b90 msgr2=0x7fc2a0160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a0099b90 0x7fc2a0160010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 -- 172.21.15.12:0/923857565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00c59c0 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a4a3a640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc298036040 tx=0x7fc298004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298043020 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29803e450 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a00c5ca0 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298009510 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.842+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a00b2ad0 con 0x7fc2a0158e30 2024-03-08T08:31:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.843+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a00749c0 con 0x7fc2a0158e30 2024-03-08T08:31:16.846 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.843+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc298046020 con 0x7fc2a0158e30 2024-03-08T08:31:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.844+0000 7fc29cff9640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.844+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc2980ace20 con 0x7fc2a0158e30 2024-03-08T08:31:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.844+0000 7fc29ffff640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.844+0000 7fc29ffff640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc290001640 tx=0x7fc290009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:16.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:16.848+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc29804d020 con 0x7fc2a0158e30 2024-03-08T08:31:16.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.889 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.955 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:16.959 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.005 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.027+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2a00b78f0 con 0x7fc2a0158e30 2024-03-08T08:31:17.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.027+0000 7fc29cff9640 1 -- 172.21.15.12:0/923857565 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc29807bb80 con 0x7fc2a0158e30 2024-03-08T08:31:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 msgr2=0x7fc28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc290001640 tx=0x7fc290009040 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 msgr2=0x7fc2a0156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc298036040 tx=0x7fc298004a40 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 shutdown_connections 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2a0158e30 0x7fc2a0156de0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a0099b90 0x7fc2a0160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 --2- 172.21.15.12:0/923857565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc2a006bef0 0x7fc2a00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 >> 172.21.15.12:0/923857565 conn(0x7fc2a00b3460 msgr2=0x7fc2a006d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 shutdown_connections 2024-03-08T08:31:17.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.029+0000 7fc2a649d640 1 -- 172.21.15.12:0/923857565 wait complete. 2024-03-08T08:31:17.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:31:17.066 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.071 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.071 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.078 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.143+0000 7f6641f16640 1 Processor -- start 2024-03-08T08:31:17.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.143+0000 7f6641f16640 1 -- start start 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f6641f16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f663c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f6641f16640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f663c0d2050 0x7f663c0b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f6641f16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c0b7250 con 0x7f663c15dbc0 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f6641f16640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c0b7520 con 0x7f663c0d2050 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f6641f16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0b7800 con 0x7f663c0b9760 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f663b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f663c15dbc0 0x7f663c0b6b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46948/0 (socket says 172.21.15.12:46948) 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.144+0000 7f663b7fe640 1 -- 172.21.15.12:0/2747484585 learned_addr learned my addr 172.21.15.12:0/2747484585 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663bfff640 1 --2- 172.21.15.12:0/2747484585 >> v2:172.21.15.156:3300/0 conn(0x7f663c0d2050 0x7f663c0b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f663c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974433813 0 0) 0x7f663c0b7250 con 0x7f663c15dbc0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6624003520 con 0x7f663c15dbc0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735293643 0 0) 0x7f6624003520 con 0x7f663c15dbc0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 msgr2=0x7f663c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f663c0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 >> v2:172.21.15.156:3300/0 conn(0x7f663c0d2050 msgr2=0x7f663c0b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 --2- 172.21.15.12:0/2747484585 >> v2:172.21.15.156:3300/0 conn(0x7f663c0d2050 0x7f663c0b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0b7ae0 con 0x7f663c15dbc0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f662c001670 con 0x7f663c15dbc0 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 >> v1:172.21.15.12:6789/0 conn(0x7f663c15dbc0 legacy=0x7f663c0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66240039b0 0x7f6624003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6624004800 0x7f6624004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f66240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0b7520 con 0x7f6624004800 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0b7800 con 0x7f66240039b0 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662400ba10 con 0x7f663c0b9760 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6624004800 0x7f6624004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663b7fe640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66240039b0 0x7f6624003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663bfff640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f66240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 -- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 msgr2=0x7f66240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f66240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 -- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66240039b0 msgr2=0x7f6624003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66240039b0 0x7f6624003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663affd640 1 -- 172.21.15.12:0/2747484585 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624003520 con 0x7f6624004800 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663bfff640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f663c0b9760 0x7f66240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.145+0000 7f663b7fe640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66240039b0 0x7f6624003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.146+0000 7f663affd640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6624004800 0x7f6624004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6630009af0 tx=0x7f6624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0e89c50cb40060 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.146+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630039070 con 0x7f6624004800 2024-03-08T08:31:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.149+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663002c6b0 con 0x7f6624004800 2024-03-08T08:31:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.149+0000 7f663a7fc640 1 -- 172.21.15.12:0/2747484585 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630009690 con 0x7f6624004800 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 -- 172.21.15.12:0/2747484585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6624004800 msgr2=0x7f6624004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 --2- 172.21.15.12:0/2747484585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6624004800 0x7f6624004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6630009af0 tx=0x7f6624005830 comp rx=0 tx=0).stop 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f663bfff640 1 -- 172.21.15.12:0/2747484585 reap_dead start 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 -- 172.21.15.12:0/2747484585 shutdown_connections 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 -- 172.21.15.12:0/2747484585 >> 172.21.15.12:0/2747484585 conn(0x7f663c072720 msgr2=0x7f663c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 -- 172.21.15.12:0/2747484585 shutdown_connections 2024-03-08T08:31:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.150+0000 7f6641f16640 1 -- 172.21.15.12:0/2747484585 wait complete. 2024-03-08T08:31:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.151+0000 7f6641f16640 1 Processor -- start 2024-03-08T08:31:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.151+0000 7f6641f16640 1 -- start start 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 0x7f663c1f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 0x7f663c0d38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c15dfa0 con 0x7f663c0ccf10 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0b80a0 con 0x7f663c1f4c50 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f6641f16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0d1b10 con 0x7f6624004800 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51032/0 (socket says 172.21.15.12:51032) 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 -- 172.21.15.12:0/3357808690 learned_addr learned my addr 172.21.15.12:0/3357808690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663b7fe640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 0x7f663c1f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663bfff640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 0x7f663c0d38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 -- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 msgr2=0x7f663c1f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 0x7f663c1f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 -- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 msgr2=0x7f663c0d38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 0x7f663c0d38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 -- 172.21.15.12:0/3357808690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c1587f0 con 0x7f663c0ccf10 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663b7fe640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 0x7f663c1f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663bfff640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 0x7f663c0d38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.152+0000 7f663affd640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6630037040 tx=0x7f6630004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.154+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630042020 con 0x7f663c0ccf10 2024-03-08T08:31:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.154+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663003a040 con 0x7f663c0ccf10 2024-03-08T08:31:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.154+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c1f3e10 con 0x7f663c0ccf10 2024-03-08T08:31:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.154+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630039070 con 0x7f663c0ccf10 2024-03-08T08:31:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.154+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c1f43d0 con 0x7f663c0ccf10 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.155+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c0749c0 con 0x7f663c0ccf10 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.155+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6630045020 con 0x7f663c0ccf10 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.156+0000 7f6623fff640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 0x7f662806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.156+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f66300ab680 con 0x7f663c0ccf10 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.156+0000 7f663b7fe640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 0x7f662806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.156+0000 7f663b7fe640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 0x7f662806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f662c001eb0 tx=0x7f662c000f40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.160+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f663004c070 con 0x7f663c0ccf10 2024-03-08T08:31:17.185 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.185 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.185 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.186 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.258 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.259 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.260 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.302 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.314 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.338+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f663c0cee10 con 0x7f663c0ccf10 2024-03-08T08:31:17.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.339+0000 7f6623fff640 1 -- 172.21.15.12:0/3357808690 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f663007a480 con 0x7f663c0ccf10 2024-03-08T08:31:17.340 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:31:17.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 msgr2=0x7f662806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 0x7f662806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f662c001eb0 tx=0x7f662c000f40 comp rx=0 tx=0).stop 2024-03-08T08:31:17.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 msgr2=0x7f663c1f4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6630037040 tx=0x7f6630004730 comp rx=0 tx=0).stop 2024-03-08T08:31:17.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 shutdown_connections 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f662806b4d0 0x7f662806d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f663c1f4c50 0x7f663c0d38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f663c0ccf10 0x7f663c1f4710 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 --2- 172.21.15.12:0/3357808690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6624004800 0x7f663c1f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 >> 172.21.15.12:0/3357808690 conn(0x7f663c072720 msgr2=0x7f663c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 shutdown_connections 2024-03-08T08:31:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.341+0000 7f6641f16640 1 -- 172.21.15.12:0/3357808690 wait complete. 2024-03-08T08:31:17.353 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:31:17.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.370 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.372 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.372 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.432 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.453+0000 7fb8e2a17640 1 Processor -- start 2024-03-08T08:31:17.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.453+0000 7fb8e2a17640 1 -- start start 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e2a17640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8dc0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e2a17640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb8dc0cdb10 0x7fb8dc099b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e2a17640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8dc0d0130 con 0x7fb8dc06bef0 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e2a17640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc0ccee0 con 0x7fb8dc0cdb10 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e2a17640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d03f0 con 0x7fb8dc0cd730 2024-03-08T08:31:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e0fb4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb8dc06bef0 0x7fb8dc15d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46964/0 (socket says 172.21.15.12:46964) 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.454+0000 7fb8e0fb4640 1 -- 172.21.15.12:0/166427012 learned_addr learned my addr 172.21.15.12:0/166427012 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791638833 0 0) 0x7fb8dc0d0130 con 0x7fb8dc06bef0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/166427012 >> v2:172.21.15.156:3300/0 conn(0x7fb8dc0cdb10 0x7fb8dc099b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8dc0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c4003520 con 0x7fb8dc06bef0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56496449 0 0) 0x7fb8c4003520 con 0x7fb8dc06bef0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 msgr2=0x7fb8dc0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8dc0cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 >> v2:172.21.15.156:3300/0 conn(0x7fb8dc0cdb10 msgr2=0x7fb8dc099b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 --2- 172.21.15.12:0/166427012 >> v2:172.21.15.156:3300/0 conn(0x7fb8dc0cdb10 0x7fb8dc099b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0d06d0 con 0x7fb8dc06bef0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fb8d40019f0 con 0x7fb8dc06bef0 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 >> v1:172.21.15.12:6789/0 conn(0x7fb8dc06bef0 legacy=0x7fb8dc15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8c40039b0 0x7fb8c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8c4004800 0x7fb8c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0ccee0 con 0x7fb8c4004800 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d03f0 con 0x7fb8c40039b0 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c400b9c0 con 0x7fb8dc0cd730 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8c4004800 0x7fb8c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8c40039b0 0x7fb8c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8e0fb4640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 -- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 msgr2=0x7fb8c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 -- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8c40039b0 msgr2=0x7fb8c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8c40039b0 0x7fb8c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8db7fe640 1 -- 172.21.15.12:0/166427012 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4003520 con 0x7fb8c4004800 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8c40039b0 0x7fb8c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.455+0000 7fb8e0fb4640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0cd730 0x7fb8c40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.456+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8c4004800 0x7fb8c4004c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009ac0 tx=0x7fb8c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7644f47c790c230c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.456+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0039070 con 0x7fb8c4004800 2024-03-08T08:31:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.456+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d002c710 con 0x7fb8c4004800 2024-03-08T08:31:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.456+0000 7fb8daffd640 1 -- 172.21.15.12:0/166427012 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0009690 con 0x7fb8c4004800 2024-03-08T08:31:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 -- 172.21.15.12:0/166427012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8c4004800 msgr2=0x7fb8c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/166427012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8c4004800 0x7fb8c4004c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009ac0 tx=0x7fb8c4005830 comp rx=0 tx=0).stop 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e0fb4640 1 -- 172.21.15.12:0/166427012 reap_dead start 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 -- 172.21.15.12:0/166427012 shutdown_connections 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 -- 172.21.15.12:0/166427012 >> 172.21.15.12:0/166427012 conn(0x7fb8dc0b3370 msgr2=0x7fb8dc0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 -- 172.21.15.12:0/166427012 shutdown_connections 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.457+0000 7fb8e2a17640 1 -- 172.21.15.12:0/166427012 wait complete. 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.458+0000 7fb8e2a17640 1 Processor -- start 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.458+0000 7fb8e2a17640 1 -- start start 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8dc155630 0x7fb8dc153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0b6460 0x7fb8dc0b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d0f70 con 0x7fb8dc155a10 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d0c90 con 0x7fb8dc155630 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e2a17640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc06d130 con 0x7fb8dc0b6460 2024-03-08T08:31:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51042/0 (socket says 172.21.15.12:51042) 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 -- 172.21.15.12:0/185391331 learned_addr learned my addr 172.21.15.12:0/185391331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8dc155630 0x7fb8dc153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8e0fb4640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0b6460 0x7fb8dc0b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 -- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0b6460 msgr2=0x7fb8dc0b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0b6460 0x7fb8dc0b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 -- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8dc155630 msgr2=0x7fb8dc153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8dc155630 0x7fb8dc153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 -- 172.21.15.12:0/185391331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc156d20 con 0x7fb8dc155a10 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.459+0000 7fb8db7fe640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009be0 tx=0x7fb8d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.460+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0039070 con 0x7fb8dc155a10 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.460+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d0043070 con 0x7fb8dc155a10 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.460+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc157000 con 0x7fb8dc155a10 2024-03-08T08:31:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.460+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00044e0 con 0x7fb8dc155a10 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.460+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc1575c0 con 0x7fb8dc155a10 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.461+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc06b000 con 0x7fb8dc155a10 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.461+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8d0003a10 con 0x7fb8dc155a10 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.461+0000 7fb8d8ff9640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 0x7fb8c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.462+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8d00ab3b0 con 0x7fb8dc155a10 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.462+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 0x7fb8c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.462+0000 7fb8dbfff640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 0x7fb8c806d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009a60 tx=0x7fb8cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.466+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb8dc06b000 con 0x7fb8dc155a10 2024-03-08T08:31:17.481 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.484 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.486 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.487 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.551 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.596 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.601 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.601 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.602 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.662 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.661+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb8dc0cf4f0 con 0x7fb8dc155a10 2024-03-08T08:31:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.662+0000 7fb8d8ff9640 1 -- 172.21.15.12:0/185391331 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb8dc0cf4f0 con 0x7fb8dc155a10 2024-03-08T08:31:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.663+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 msgr2=0x7fb8c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 0x7fb8c806d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009a60 tx=0x7fb8cc008040 comp rx=0 tx=0).stop 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 msgr2=0x7fb8dc0c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009be0 tx=0x7fb8d0004730 comp rx=0 tx=0).stop 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 shutdown_connections 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb8c806b500 0x7fb8c806d9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8dc0b6460 0x7fb8dc0b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8dc155a10 0x7fb8dc0c7a90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 --2- 172.21.15.12:0/185391331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8dc155630 0x7fb8dc153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 >> 172.21.15.12:0/185391331 conn(0x7fb8dc0b3370 msgr2=0x7fb8dc0b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 shutdown_connections 2024-03-08T08:31:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:17.664+0000 7fb8e2a17640 1 -- 172.21.15.12:0/185391331 wait complete. 2024-03-08T08:31:17.667 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.675 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-03-08T08:31:17.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:17.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-08T08:31:17.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[51735]: fuse finished with error 0 and tester_r 0 2024-03-08T08:31:17.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:31:17.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:17.705 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-08T08:31:17.710 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.712 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.715 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.773 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.782 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.828 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.830 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.831 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.850 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.869 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:17.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:17 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:17.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.895 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.895 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.913 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:17.982 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.007 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.008 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.122 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.123 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.123 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.123 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.189 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.228 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.234 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.235 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.236 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.300 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.348 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.349 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.350 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.352 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.372 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.385 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:18.411 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:18.433 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.436 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.436 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.437 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.480 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.491 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.505 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:18.545 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:18.551 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.552 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.552 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.554 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.609 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.622 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:18.659 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:18.664 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.665 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.665 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.667 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.721 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.725 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.732 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.733 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:18.775 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:18.776 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.777 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.779 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.858 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:18.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.891 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:18.899 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:18.975 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.006 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.007 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.068 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.073 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.073 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.079 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.093 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.124 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.185 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.187 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.187 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.187 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.197 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.211 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.261 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.261 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.261 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.304 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.315 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.328 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.368 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.373 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.373 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.374 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.434 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.448 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.484 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.488 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.489 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.552 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.566 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.598 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.664 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.669 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.683 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.711 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.714 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.717 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.776 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.784 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.796 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.830 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.833 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.834 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.852 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.896 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.896 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.896 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.897 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:19.908 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:19.915 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:19.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:19 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:19.984 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.023 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.076 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.124 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.124 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.125 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.125 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.136 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.190 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.230 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.237 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.244 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.302 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.350 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.351 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.351 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.354 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.364 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.413 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.435 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.438 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.439 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.439 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.449 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.482 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.547 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.554 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.554 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.554 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.556 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.567 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.661 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.666 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.667 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.667 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.669 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.682 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.734 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.777 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.777 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.779 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.781 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.790 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.893 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:20.901 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:20.906 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:20.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:20 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:20.960 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.022 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.070 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.074 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.075 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.083 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.126 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.187 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.188 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.188 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.189 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.204 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.263 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.263 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.263 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.276 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.306 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.371 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.375 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.375 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.375 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.388 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.486 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.488 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.489 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.491 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.501 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.600 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.605 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.605 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.606 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.618 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.666 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.714 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.716 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.718 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.726 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.778 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.833 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.835 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.835 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.848 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.853 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.898 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.899 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.899 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:21.913 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:21.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:21 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:21.986 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.012 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.024 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.078 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.124 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.126 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.127 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.140 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.192 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.238 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.238 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.239 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.250 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.304 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.356 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.369 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.415 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.441 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.441 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.441 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.454 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.484 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.549 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.555 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.556 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.558 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.568 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.663 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.668 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.668 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.668 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.681 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.722 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.735 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.779 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.781 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.783 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.792 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.894 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.895 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.903 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.909 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:22.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:22.962 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:22.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:22.975 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.009 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.013 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.077 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.077 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.090 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.128 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.189 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.190 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.190 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.203 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.265 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.265 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.277 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.306 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.373 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.377 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.390 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.488 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.491 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.492 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.504 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.602 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.607 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.607 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.619 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.668 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.719 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.719 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.731 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.779 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.835 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.838 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.850 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.855 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.901 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.901 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:23.912 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:23.918 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:23.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:23 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:23.988 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.013 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.014 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.026 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.080 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.127 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.129 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.139 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.194 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.240 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.241 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.253 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.306 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.353 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.354 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.359 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.367 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.418 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.443 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.443 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.456 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.486 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.551 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.558 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.560 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.574 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.666 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.669 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.670 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.682 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.725 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.737 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.783 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.785 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.796 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:24.896 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:24.905 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:24.908 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:24 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.011 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.016 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.024 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.079 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.090 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.130 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.191 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.191 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.202 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.267 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.278 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.309 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.373 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.379 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.392 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.494 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.507 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.604 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.609 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.621 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.669 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.721 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.725 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.733 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.781 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.837 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.850 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.858 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.903 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:25.916 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:25.921 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:25.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:25 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.015 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.028 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.082 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.130 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.143 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.197 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.243 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.254 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.308 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.355 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.360 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.369 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.420 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.445 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.457 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.488 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.553 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.560 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.574 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.668 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.672 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.687 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.728 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.787 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.800 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:26.907 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:26.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:26.920 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:26.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.018 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.032 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.132 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.145 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.193 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.206 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.311 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.323 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.375 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.388 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.492 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.505 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.606 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.620 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.671 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.683 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.725 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.783 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.796 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.860 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.872 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.923 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:27.935 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:27.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:27.992 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:27.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.004 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.085 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.097 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.198 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.212 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.310 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.323 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.422 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.434 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.490 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.502 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.555 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.566 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.669 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.680 INFO:tasks.cephfs.mount:Terminating background process 2024-03-08T08:31:28.722 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-08T08:31:28.742 INFO:teuthology.orchestra.run.smithi156.stderr:open_background: keeping file open 2024-03-08T08:31:28.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-08T08:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T08:31:28.756 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:31:28.756 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:28.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:28.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.925+0000 7feab2eaa640 1 Processor -- start 2024-03-08T08:31:28.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.925+0000 7feab2eaa640 1 -- start start 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab2eaa640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feaac0aeb30 0x7feaac15b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab2eaa640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 0x7feaac0ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab2eaa640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0cbe00 con 0x7feaac0aef10 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab2eaa640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0cc0e0 con 0x7feaac0aeb30 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab2eaa640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0cc3c0 con 0x7feaac15e9a0 2024-03-08T08:31:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab0c46640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feaac0aeb30 0x7feaac15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab0c46640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feaac0aeb30 0x7feaac15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39082/0 (socket says 172.21.15.12:39082) 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab0c46640 1 -- 172.21.15.12:0/1660441735 learned_addr learned my addr 172.21.15.12:0/1660441735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.926+0000 7feab1447640 1 --2- 172.21.15.12:0/1660441735 >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 0x7feaac0ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab0c46640 1 -- 172.21.15.12:0/1660441735 >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 msgr2=0x7feaac0ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab0c46640 1 --2- 172.21.15.12:0/1660441735 >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 0x7feaac0ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab0c46640 1 -- 172.21.15.12:0/1660441735 >> v1:172.21.15.156:6789/0 conn(0x7feaac0aef10 legacy=0x7feaac071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab0c46640 1 -- 172.21.15.12:0/1660441735 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0cc6a0 con 0x7feaac0aeb30 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab1447640 1 --2- 172.21.15.12:0/1660441735 >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 0x7feaac0ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.927+0000 7feab0c46640 1 --2- 172.21.15.12:0/1660441735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0aeb30 0x7feaac15b070 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009870 tx=0x7feaa002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6847144484237d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:28.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feaab7fe640 1 -- 172.21.15.12:0/1660441735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa002e070 con 0x7feaac0aeb30 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feaab7fe640 1 -- 172.21.15.12:0/1660441735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0037070 con 0x7feaac0aeb30 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feaab7fe640 1 -- 172.21.15.12:0/1660441735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0031a00 con 0x7feaac0aeb30 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 -- 172.21.15.12:0/1660441735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0aeb30 msgr2=0x7feaac15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 --2- 172.21.15.12:0/1660441735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0aeb30 0x7feaac15b070 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009870 tx=0x7feaa002b880 comp rx=0 tx=0).stop 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 -- 172.21.15.12:0/1660441735 shutdown_connections 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 --2- 172.21.15.12:0/1660441735 >> v2:172.21.15.156:3300/0 conn(0x7feaac15e9a0 0x7feaac0ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 --2- 172.21.15.12:0/1660441735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0aeb30 0x7feaac15b070 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 -- 172.21.15.12:0/1660441735 >> 172.21.15.12:0/1660441735 conn(0x7feaac06abd0 msgr2=0x7feaac06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 -- 172.21.15.12:0/1660441735 shutdown_connections 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.928+0000 7feab2eaa640 1 -- 172.21.15.12:0/1660441735 wait complete. 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.929+0000 7feab2eaa640 1 Processor -- start 2024-03-08T08:31:28.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.929+0000 7feab2eaa640 1 -- start start 2024-03-08T08:31:28.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaac0aeb30 0x7feaac1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feaac0aef10 0x7feaac0ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0725e0 con 0x7feaac0cf020 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac072460 con 0x7feaac0aeb30 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab2eaa640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0714c0 con 0x7feaac0aef10 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab1447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab1447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39092/0 (socket says 172.21.15.12:39092) 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.930+0000 7feab1447640 1 -- 172.21.15.12:0/3542369488 learned_addr learned my addr 172.21.15.12:0/3542369488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 -- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feaac0aef10 msgr2=0x7feaac0ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feaac0aef10 0x7feaac0ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 -- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaac0aeb30 msgr2=0x7feaac1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaac0aeb30 0x7feaac1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 -- 172.21.15.12:0/3542369488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac13d3f0 con 0x7feaac0cf020 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab1447640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feaa400cd80 tx=0x7feaa400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa400f680 con 0x7feaac0cf020 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa400f840 con 0x7feaac0cf020 2024-03-08T08:31:28.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac13d6d0 con 0x7feaac0cf020 2024-03-08T08:31:28.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa400fc30 con 0x7feaac0cf020 2024-03-08T08:31:28.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.931+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac13dc90 con 0x7feaac0cf020 2024-03-08T08:31:28.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.932+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac13de90 con 0x7feaac0cf020 2024-03-08T08:31:28.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.933+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaa40213f0 con 0x7feaac0cf020 2024-03-08T08:31:28.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.933+0000 7feaa97fa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 0x7fea9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:28.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.933+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feaa408afd0 con 0x7feaac0cf020 2024-03-08T08:31:28.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.933+0000 7feab0c46640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 0x7fea9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:28.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.934+0000 7feab0c46640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 0x7fea9006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feaa0036000 tx=0x7feaa0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:28.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:28.938+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feaac13de90 con 0x7feaac0cf020 2024-03-08T08:31:29.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.123+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaac0749c0 con 0x7feaac0cf020 2024-03-08T08:31:29.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.124+0000 7feaa97fa640 1 -- 172.21.15.12:0/3542369488 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16859 (secure 0 0 0) 0x7feaa4014030 con 0x7feaac0cf020 2024-03-08T08:31:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:29.126 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":37,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:55.107876+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.35","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.126+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 msgr2=0x7fea9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.126+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 0x7fea9006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feaa0036000 tx=0x7feaa0030000 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.126+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 msgr2=0x7feaac13f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.126+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feaa400cd80 tx=0x7feaa400a650 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 shutdown_connections 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fea9006b4d0 0x7fea9006d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feaac0cf020 0x7feaac13f580 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feaac0aef10 0x7feaac0ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 --2- 172.21.15.12:0/3542369488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaac0aeb30 0x7feaac1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 >> 172.21.15.12:0/3542369488 conn(0x7feaac06abd0 msgr2=0x7feaac153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 shutdown_connections 2024-03-08T08:31:29.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.127+0000 7feab2eaa640 1 -- 172.21.15.12:0/3542369488 wait complete. 2024-03-08T08:31:29.139 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:30:09.998602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:30:11.090210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:31:29.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:31:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.242+0000 7fde24233640 1 Processor -- start 2024-03-08T08:31:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 -- start start 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde1c15dbc0 0x7fde1c0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fde1c0b7080 0x7fde1c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c0b7520 con 0x7fde1c0d2050 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0b7800 con 0x7fde1c15dbc0 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde24233640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0b7ae0 con 0x7fde1c0b7080 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde21fcf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde1c15dbc0 0x7fde1c0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde227d0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fde1c0d2050 0x7fde1c0b85c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34694/0 (socket says 172.21.15.12:34694) 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.243+0000 7fde227d0640 1 -- 172.21.15.12:0/92202003 learned_addr learned my addr 172.21.15.12:0/92202003 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde217ce640 1 --2- 172.21.15.12:0/92202003 >> v2:172.21.15.156:3300/0 conn(0x7fde1c0b7080 0x7fde1c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde21fcf640 1 -- 172.21.15.12:0/92202003 >> v2:172.21.15.156:3300/0 conn(0x7fde1c0b7080 msgr2=0x7fde1c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde21fcf640 1 --2- 172.21.15.12:0/92202003 >> v2:172.21.15.156:3300/0 conn(0x7fde1c0b7080 0x7fde1c0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde21fcf640 1 -- 172.21.15.12:0/92202003 >> v1:172.21.15.12:6789/0 conn(0x7fde1c0d2050 legacy=0x7fde1c0b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde21fcf640 1 -- 172.21.15.12:0/92202003 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0b7dc0 con 0x7fde1c15dbc0 2024-03-08T08:31:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.244+0000 7fde20fcd640 1 -- 172.21.15.12:0/92202003 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699235995 0 0) 0x7fde1c0b7520 con 0x7fde1c0d2050 2024-03-08T08:31:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde21fcf640 1 --2- 172.21.15.12:0/92202003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0b6b40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0ce0b0 tx=0x7fde1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24040396f6fff6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde20fcd640 1 -- 172.21.15.12:0/92202003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18035020 con 0x7fde1c15dbc0 2024-03-08T08:31:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde20fcd640 1 -- 172.21.15.12:0/92202003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18031830 con 0x7fde1c15dbc0 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde20fcd640 1 -- 172.21.15.12:0/92202003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18031c00 con 0x7fde1c15dbc0 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 -- 172.21.15.12:0/92202003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 msgr2=0x7fde1c0b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 --2- 172.21.15.12:0/92202003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0b6b40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0ce0b0 tx=0x7fde1802b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 -- 172.21.15.12:0/92202003 shutdown_connections 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 --2- 172.21.15.12:0/92202003 >> v2:172.21.15.156:3300/0 conn(0x7fde1c0b7080 0x7fde1c0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 --2- 172.21.15.12:0/92202003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0b6b40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 -- 172.21.15.12:0/92202003 >> 172.21.15.12:0/92202003 conn(0x7fde1c072720 msgr2=0x7fde1c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 -- 172.21.15.12:0/92202003 shutdown_connections 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.245+0000 7fde24233640 1 -- 172.21.15.12:0/92202003 wait complete. 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.246+0000 7fde24233640 1 Processor -- start 2024-03-08T08:31:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.246+0000 7fde24233640 1 -- start start 2024-03-08T08:31:29.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1c0b7080 0x7fde1c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 0x7fde1c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0b8cd0 con 0x7fde1c15dbc0 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c15dfa0 con 0x7fde1c0d2050 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde24233640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0b8e50 con 0x7fde1c0b7080 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde227d0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde227d0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39108/0 (socket says 172.21.15.12:39108) 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde227d0640 1 -- 172.21.15.12:0/3495650694 learned_addr learned my addr 172.21.15.12:0/3495650694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde217ce640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 0x7fde1c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.247+0000 7fde227d0640 1 -- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1c0b7080 msgr2=0x7fde1c0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde21fcf640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1c0b7080 0x7fde1c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde227d0640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1c0b7080 0x7fde1c0cb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde227d0640 1 -- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 msgr2=0x7fde1c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde227d0640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 0x7fde1c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde227d0640 1 -- 172.21.15.12:0/3495650694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0cbb70 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde217ce640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 0x7fde1c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde227d0640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fde1400cb20 tx=0x7fde1400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400aca0 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14012400 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400ae60 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c0cbe50 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.248+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c0cc410 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.249+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c0749c0 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.249+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde140127f0 con 0x7fde1c15dbc0 2024-03-08T08:31:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.250+0000 7fde127fc640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 0x7fde0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.250+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fde1408a940 con 0x7fde1c15dbc0 2024-03-08T08:31:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.250+0000 7fde21fcf640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 0x7fde0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.250+0000 7fde21fcf640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 0x7fde0006d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fde18036000 tx=0x7fde18008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.254+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fde14059730 con 0x7fde1c15dbc0 2024-03-08T08:31:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.432+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde1c0b6b00 con 0x7fde1c15dbc0 2024-03-08T08:31:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.432+0000 7fde127fc640 1 -- 172.21.15.12:0/3495650694 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde14059030 con 0x7fde1c15dbc0 2024-03-08T08:31:29.433 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:31:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.434+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 msgr2=0x7fde0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.434+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 0x7fde0006d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fde18036000 tx=0x7fde18008030 comp rx=0 tx=0).stop 2024-03-08T08:31:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.434+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 msgr2=0x7fde1c0c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.434+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fde1400cb20 tx=0x7fde1400a060 comp rx=0 tx=0).stop 2024-03-08T08:31:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 shutdown_connections 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fde0006b4d0 0x7fde0006d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1c15dbc0 0x7fde1c0c9110 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1c0d2050 0x7fde1c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 --2- 172.21.15.12:0/3495650694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1c0b7080 0x7fde1c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 >> 172.21.15.12:0/3495650694 conn(0x7fde1c072720 msgr2=0x7fde1c0d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 shutdown_connections 2024-03-08T08:31:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.435+0000 7fde24233640 1 -- 172.21.15.12:0/3495650694 wait complete. 2024-03-08T08:31:29.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:31:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.549+0000 7fc442f08640 1 Processor -- start 2024-03-08T08:31:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.549+0000 7fc442f08640 1 -- start start 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc442f08640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc442f08640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc442f08640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c0d1d90 con 0x7fc43c161260 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc442f08640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c0b9ac0 con 0x7fc43c0b6b30 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc442f08640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0d2070 con 0x7fc43c06c160 2024-03-08T08:31:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc4414a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc43bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc43bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:39262/0 (socket says 172.21.15.12:39262) 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc4414a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40776/0 (socket says 172.21.15.12:40776) 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc43bfff640 1 -- 172.21.15.12:0/3149494469 learned_addr learned my addr 172.21.15.12:0/3149494469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.550+0000 7fc440ca4640 1 --1- 172.21.15.12:0/3149494469 >> v1:172.21.15.156:6789/0 conn(0x7fc43c161260 0x7fc43c0b93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:35240/0 (socket says 172.21.15.12:35240) 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.551+0000 7fc4414a5640 1 -- 172.21.15.12:0/3149494469 >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 msgr2=0x7fc43c0b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.551+0000 7fc4414a5640 1 --2- 172.21.15.12:0/3149494469 >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.551+0000 7fc4414a5640 1 -- 172.21.15.12:0/3149494469 >> v1:172.21.15.156:6789/0 conn(0x7fc43c161260 legacy=0x7fc43c0b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.551+0000 7fc4414a5640 1 -- 172.21.15.12:0/3149494469 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c0d2350 con 0x7fc43c06c160 2024-03-08T08:31:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.551+0000 7fc43bfff640 1 --2- 172.21.15.12:0/3149494469 >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc4414a5640 1 --2- 172.21.15.12:0/3149494469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0097a0 tx=0x7fc42c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b22610f5f9b18d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc43b7fe640 1 -- 172.21.15.12:0/3149494469 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c035020 con 0x7fc43c06c160 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc43b7fe640 1 -- 172.21.15.12:0/3149494469 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c031830 con 0x7fc43c06c160 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc43b7fe640 1 -- 172.21.15.12:0/3149494469 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c031c00 con 0x7fc43c06c160 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 -- 172.21.15.12:0/3149494469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 msgr2=0x7fc43c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 --2- 172.21.15.12:0/3149494469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0097a0 tx=0x7fc42c02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 -- 172.21.15.12:0/3149494469 shutdown_connections 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 --2- 172.21.15.12:0/3149494469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c06c160 0x7fc43c06c5f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 --2- 172.21.15.12:0/3149494469 >> v2:172.21.15.156:3300/0 conn(0x7fc43c0b6b30 0x7fc43c0b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 -- 172.21.15.12:0/3149494469 >> 172.21.15.12:0/3149494469 conn(0x7fc43c072930 msgr2=0x7fc43c0d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 -- 172.21.15.12:0/3149494469 shutdown_connections 2024-03-08T08:31:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.552+0000 7fc442f08640 1 -- 172.21.15.12:0/3149494469 wait complete. 2024-03-08T08:31:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 Processor -- start 2024-03-08T08:31:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 -- start start 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43c0b6b30 0x7fc43c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c161260 0x7fc43c0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0d2e30 con 0x7fc43c06c160 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0b2ec0 con 0x7fc43c0b6b30 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.554+0000 7fc442f08640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0d2fb0 con 0x7fc43c161260 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39110/0 (socket says 172.21.15.12:39110) 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 -- 172.21.15.12:0/1526824017 learned_addr learned my addr 172.21.15.12:0/1526824017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc43bfff640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43c0b6b30 0x7fc43c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc4414a5640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c161260 0x7fc43c0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 -- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c161260 msgr2=0x7fc43c0c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c161260 0x7fc43c0c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 -- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43c0b6b30 msgr2=0x7fc43c0c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43c0b6b30 0x7fc43c0c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 -- 172.21.15.12:0/1526824017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c0cbf60 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc440ca4640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c730 tx=0x7fc43400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434017070 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43400ce40 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c0cc240 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43400a860 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.555+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c0cc800 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.556+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c0749c0 con 0x7fc43c06c160 2024-03-08T08:31:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.557+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc43401a070 con 0x7fc43c06c160 2024-03-08T08:31:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.557+0000 7fc4397fa640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 0x7fc42806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.557+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc434089770 con 0x7fc43c06c160 2024-03-08T08:31:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.557+0000 7fc43bfff640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 0x7fc42806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.558+0000 7fc43bfff640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 0x7fc42806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0b6320 tx=0x7fc430009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.562+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc434057fb0 con 0x7fc43c06c160 2024-03-08T08:31:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.738+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc43c0cc9d0 con 0x7fc43c06c160 2024-03-08T08:31:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.739+0000 7fc4397fa640 1 -- 172.21.15.12:0/1526824017 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc43405c050 con 0x7fc43c06c160 2024-03-08T08:31:29.740 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 msgr2=0x7fc42806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 0x7fc42806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0b6320 tx=0x7fc430009240 comp rx=0 tx=0).stop 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 msgr2=0x7fc43c14fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c730 tx=0x7fc43400cbf0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 shutdown_connections 2024-03-08T08:31:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc42806b4d0 0x7fc42806d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc43c161260 0x7fc43c0c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43c0b6b30 0x7fc43c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 --2- 172.21.15.12:0/1526824017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc43c06c160 0x7fc43c14fec0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 >> 172.21.15.12:0/1526824017 conn(0x7fc43c072930 msgr2=0x7fc43c0b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 shutdown_connections 2024-03-08T08:31:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.741+0000 7fc442f08640 1 -- 172.21.15.12:0/1526824017 wait complete. 2024-03-08T08:31:29.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:31:29.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.855+0000 7fb4a9b08640 1 Processor -- start 2024-03-08T08:31:29.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.855+0000 7fb4a9b08640 1 -- start start 2024-03-08T08:31:29.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a9b08640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb4a40cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a9b08640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40cdf60 con 0x7fb4a4099b90 2024-03-08T08:31:29.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a9b08640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40cf700 con 0x7fb4a415bb90 2024-03-08T08:31:29.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a9b08640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40cd590 con 0x7fb4a40cc390 2024-03-08T08:31:29.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb4a4099b90 0x7fb4a40c53d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34696/0 (socket says 172.21.15.12:34696) 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.856+0000 7fb4a3fff640 1 -- 172.21.15.12:0/729015215 learned_addr learned my addr 172.21.15.12:0/729015215 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a2ffd640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb4a40cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167353935 0 0) 0x7fb4a40cdf60 con 0x7fb4a4099b90 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c003520 con 0x7fb4a4099b90 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122322271 0 0) 0x7fb48c003520 con 0x7fb4a4099b90 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 msgr2=0x7fb4a40cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb4a40cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 >> v1:172.21.15.156:6789/0 conn(0x7fb4a415bb90 legacy=0x7fb4a40d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40cc980 con 0x7fb4a4099b90 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fb49c0019f0 con 0x7fb4a4099b90 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 >> v1:172.21.15.12:6789/0 conn(0x7fb4a4099b90 legacy=0x7fb4a40c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb48c0039e0 0x7fb48c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c004390 0x7fb48c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb48c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40cd590 con 0x7fb48c0039e0 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c00b9c0 con 0x7fb48c004390 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c00bbc0 con 0x7fb4a40cc390 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb48c0039e0 0x7fb48c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a2ffd640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c004390 0x7fb48c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a3fff640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb48c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 -- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 msgr2=0x7fb48c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb48c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 -- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c004390 msgr2=0x7fb48c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c004390 0x7fb48c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a37fe640 1 -- 172.21.15.12:0/729015215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c003520 con 0x7fb48c0039e0 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.857+0000 7fb4a3fff640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a40cc390 0x7fb48c0050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.858+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb48c0039e0 0x7fb48c003dc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb494001860 tx=0x7fb48c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744237877d90232a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.858+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49402f050 con 0x7fb48c0039e0 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.858+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494033650 con 0x7fb48c0039e0 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.858+0000 7fb4a27fc640 1 -- 172.21.15.12:0/729015215 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494001430 con 0x7fb48c0039e0 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 -- 172.21.15.12:0/729015215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb48c0039e0 msgr2=0x7fb48c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/729015215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb48c0039e0 0x7fb48c003dc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb494001860 tx=0x7fb48c005830 comp rx=0 tx=0).stop 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a3fff640 1 -- 172.21.15.12:0/729015215 reap_dead start 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 -- 172.21.15.12:0/729015215 shutdown_connections 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 -- 172.21.15.12:0/729015215 >> 172.21.15.12:0/729015215 conn(0x7fb4a40b3370 msgr2=0x7fb4a40b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 -- 172.21.15.12:0/729015215 shutdown_connections 2024-03-08T08:31:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.859+0000 7fb4a9b08640 1 -- 172.21.15.12:0/729015215 wait complete. 2024-03-08T08:31:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.860+0000 7fb4a9b08640 1 Processor -- start 2024-03-08T08:31:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.860+0000 7fb4a9b08640 1 -- start start 2024-03-08T08:31:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 0x7fb4a40ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a4153020 0x7fb4a414fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40cd220 con 0x7fb4a406c580 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40ccf40 con 0x7fb48c0039e0 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a9b08640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a415bf70 con 0x7fb4a4153020 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39138/0 (socket says 172.21.15.12:39138) 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 -- 172.21.15.12:0/3944244380 learned_addr learned my addr 172.21.15.12:0/3944244380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a2ffd640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a4153020 0x7fb4a414fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 0x7fb4a40ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 -- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a4153020 msgr2=0x7fb4a414fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a4153020 0x7fb4a414fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 -- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 msgr2=0x7fb4a40ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 0x7fb4a40ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 -- 172.21.15.12:0/3944244380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a41534a0 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 0x7fb4a40ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.861+0000 7fb4a3fff640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4073300 tx=0x7fb49c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.862+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00eda0 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.862+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c014430 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.862+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c014820 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.862+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a40ca950 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.862+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a406bd80 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.863+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a40749c0 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.863+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb49c0149c0 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.863+0000 7fb48bfff640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 0x7fb49006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.863+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb49c08a0d0 con 0x7fb4a406c580 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.863+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 0x7fb49006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.864+0000 7fb4a37fe640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 0x7fb49006d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb494001860 tx=0x7fb494031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:29.868+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb49c058ed0 con 0x7fb4a406c580 2024-03-08T08:31:30.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.046+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4a40b4210 con 0x7fb4a406c580 2024-03-08T08:31:30.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.047+0000 7fb48bfff640 1 -- 172.21.15.12:0/3944244380 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb49c0587d0 con 0x7fb4a406c580 2024-03-08T08:31:30.049 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:31:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 msgr2=0x7fb49006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 0x7fb49006d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb494001860 tx=0x7fb494031000 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 msgr2=0x7fb4a409a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4073300 tx=0x7fb49c00e530 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 shutdown_connections 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb49006b500 0x7fb49006d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a406c580 0x7fb4a409a540 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb4a4153020 0x7fb4a414fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 --2- 172.21.15.12:0/3944244380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb48c0039e0 0x7fb4a40ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 >> 172.21.15.12:0/3944244380 conn(0x7fb4a40b3370 msgr2=0x7fb4a406f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 shutdown_connections 2024-03-08T08:31:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.049+0000 7fb4a9b08640 1 -- 172.21.15.12:0/3944244380 wait complete. 2024-03-08T08:31:30.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T08:31:30.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.162+0000 7fb2317e2640 1 Processor -- start 2024-03-08T08:31:30.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.162+0000 7fb2317e2640 1 -- start start 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb2317e2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb22c0d1110 0x7fb22c0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb2317e2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb22c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb2317e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c15e1a0 con 0x7fb22c15ddc0 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb2317e2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22c0b9030 con 0x7fb22c0d1110 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb2317e2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0d1d20 con 0x7fb22c0d31c0 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb22affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb22c15ddc0 0x7fb22c0d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34712/0 (socket says 172.21.15.12:34712) 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb22affd640 1 -- 172.21.15.12:0/3597539966 learned_addr learned my addr 172.21.15.12:0/3597539966 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732383370 0 0) 0x7fb22c15e1a0 con 0x7fb22c15ddc0 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb218003520 con 0x7fb22c15ddc0 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3597539966 >> v2:172.21.15.156:3300/0 conn(0x7fb22c0d1110 0x7fb22c0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb22c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2829255564 0 0) 0x7fb218003520 con 0x7fb22c15ddc0 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 msgr2=0x7fb22c0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.163+0000 7fb229ffb640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb22c0b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 >> v2:172.21.15.156:3300/0 conn(0x7fb22c0d1110 msgr2=0x7fb22c0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 --2- 172.21.15.12:0/3597539966 >> v2:172.21.15.156:3300/0 conn(0x7fb22c0d1110 0x7fb22c0b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0b8790 con 0x7fb22c15ddc0 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fb224001670 con 0x7fb22c15ddc0 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 >> v1:172.21.15.12:6789/0 conn(0x7fb22c15ddc0 legacy=0x7fb22c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb2180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb218003fe0 0x7fb218004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb2180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0d1d20 con 0x7fb218004d40 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21800ba70 con 0x7fb218003fe0 2024-03-08T08:31:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21800bc40 con 0x7fb22c0d31c0 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb2180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22affd640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb2180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb218003fe0 0x7fb218004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 -- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 msgr2=0x7fb2180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb2180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 -- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb218003fe0 msgr2=0x7fb218004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb218003fe0 0x7fb218004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22b7fe640 1 -- 172.21.15.12:0/3597539966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218003520 con 0x7fb218004d40 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb218003fe0 0x7fb218004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.164+0000 7fb22affd640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0d31c0 0x7fb2180039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb2180051c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007360 tx=0x7fb218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786f0f6c2500bea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c039070 con 0x7fb218004d40 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c02ccd0 con 0x7fb218004d40 2024-03-08T08:31:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb229ffb640 1 -- 172.21.15.12:0/3597539966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c009be0 con 0x7fb218004d40 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 -- 172.21.15.12:0/3597539966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 msgr2=0x7fb2180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3597539966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb2180051c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007360 tx=0x7fb218005830 comp rx=0 tx=0).stop 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb22b7fe640 1 -- 172.21.15.12:0/3597539966 reap_dead start 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 -- 172.21.15.12:0/3597539966 shutdown_connections 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 -- 172.21.15.12:0/3597539966 >> 172.21.15.12:0/3597539966 conn(0x7fb22c072720 msgr2=0x7fb22c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 -- 172.21.15.12:0/3597539966 shutdown_connections 2024-03-08T08:31:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.165+0000 7fb2317e2640 1 -- 172.21.15.12:0/3597539966 wait complete. 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 Processor -- start 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 -- start start 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb22c1f4940 0x7fb22c0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 0x7fb22c15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b9980 con 0x7fb218004d40 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b8d50 con 0x7fb22c1f4940 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.167+0000 7fb2317e2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b7120 con 0x7fb22c0cd4c0 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39148/0 (socket says 172.21.15.12:39148) 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 -- 172.21.15.12:0/3977403676 learned_addr learned my addr 172.21.15.12:0/3977403676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb22c1f4940 0x7fb22c0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 0x7fb22c15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 -- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 msgr2=0x7fb22c15eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 0x7fb22c15eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 -- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb22c1f4940 msgr2=0x7fb22c0ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb22c1f4940 0x7fb22c0ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 -- 172.21.15.12:0/3977403676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0cad20 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22b7fe640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 0x7fb22c15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb22affd640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb22400eb20 tx=0x7fb224005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224019070 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2240049f0 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c0cb000 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224004da0 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.168+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c0cb5c0 con 0x7fb218004d40 2024-03-08T08:31:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.169+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c06d070 con 0x7fb218004d40 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.170+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb22401c070 con 0x7fb218004d40 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.170+0000 7fb2237fe640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 0x7fb20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.170+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb224089ac0 con 0x7fb218004d40 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.170+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 0x7fb20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.170+0000 7fb22a7fc640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 0x7fb20c06d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0cdb60 tx=0x7fb214008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.174+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb22c06d070 con 0x7fb218004d40 2024-03-08T08:31:30.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.368+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb22c0b6ee0 con 0x7fb218004d40 2024-03-08T08:31:30.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.368+0000 7fb2237fe640 1 -- 172.21.15.12:0/3977403676 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb22c0b6ee0 con 0x7fb218004d40 2024-03-08T08:31:30.370 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 msgr2=0x7fb20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 0x7fb20c06d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0cdb60 tx=0x7fb214008040 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 msgr2=0x7fb22c1f6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb22400eb20 tx=0x7fb224005e10 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 shutdown_connections 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fb20c06b4d0 0x7fb20c06d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb22c0cd4c0 0x7fb22c15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb22c1f4940 0x7fb22c0ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 --2- 172.21.15.12:0/3977403676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb218004d40 0x7fb22c1f6600 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 >> 172.21.15.12:0/3977403676 conn(0x7fb22c072720 msgr2=0x7fb22c15d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:30.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.370+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 shutdown_connections 2024-03-08T08:31:30.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.371+0000 7fb2317e2640 1 -- 172.21.15.12:0/3977403676 wait complete. 2024-03-08T08:31:30.382 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-08T08:31:30.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:30.383 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-08T08:31:30.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[43673]: fuse finished with error 0 and tester_r 0 2024-03-08T08:31:30.415 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:31:30.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:30.416 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-08T08:31:30.507 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:31:30.507 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:30.522 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=208 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-08T08:31:30.522 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:31:30.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:30.522 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-08T08:31:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:31:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-08T08:31:30.577 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:31:30.577 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:30.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-08T08:31:30.594 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:31:30.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:30.594 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-08T08:31:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:31:30.650 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-08T08:31:30.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:31:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.750+0000 7f23603e9640 1 Processor -- start 2024-03-08T08:31:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.750+0000 7f23603e9640 1 -- start start 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f23603e9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f235815f200 0x7f235815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f23603e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 0x7f235815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f23603e9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580b2df0 con 0x7f2358160370 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f23603e9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2358099b90 con 0x7f235815f200 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f23603e9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580cff50 con 0x7f23580b2830 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f235e185640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f235815f200 0x7f235815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f235e986640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2358160370 0x7f23580cc480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34718/0 (socket says 172.21.15.12:34718) 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f235e986640 1 -- 172.21.15.12:0/3608364886 learned_addr learned my addr 172.21.15.12:0/3608364886 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.751+0000 7f235d984640 1 --2- 172.21.15.12:0/3608364886 >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 0x7f235815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235e185640 1 -- 172.21.15.12:0/3608364886 >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 msgr2=0x7f235815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235e185640 1 --2- 172.21.15.12:0/3608364886 >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 0x7f235815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235e185640 1 -- 172.21.15.12:0/3608364886 >> v1:172.21.15.12:6789/0 conn(0x7f2358160370 legacy=0x7f23580cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235e185640 1 -- 172.21.15.12:0/3608364886 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580d01a0 con 0x7f235815f200 2024-03-08T08:31:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235d984640 1 --2- 172.21.15.12:0/3608364886 >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 0x7f235815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235e185640 1 --2- 172.21.15.12:0/3608364886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f235815f8f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f23580b40a0 tx=0x7f234802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2695c68f4be73d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.752+0000 7f235d183640 1 -- 172.21.15.12:0/3608364886 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348039070 con 0x7f235815f200 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f235d183640 1 -- 172.21.15.12:0/3608364886 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234802cc80 con 0x7f235815f200 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f235d183640 1 -- 172.21.15.12:0/3608364886 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348034660 con 0x7f235815f200 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 -- 172.21.15.12:0/3608364886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 msgr2=0x7f235815f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 --2- 172.21.15.12:0/3608364886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f235815f8f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f23580b40a0 tx=0x7f234802bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 -- 172.21.15.12:0/3608364886 shutdown_connections 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 --2- 172.21.15.12:0/3608364886 >> v2:172.21.15.156:3300/0 conn(0x7f23580b2830 0x7f235815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 --2- 172.21.15.12:0/3608364886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f235815f8f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 -- 172.21.15.12:0/3608364886 >> 172.21.15.12:0/3608364886 conn(0x7f23580b3460 msgr2=0x7f23580b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 -- 172.21.15.12:0/3608364886 shutdown_connections 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.753+0000 7f23603e9640 1 -- 172.21.15.12:0/3608364886 wait complete. 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.754+0000 7f23603e9640 1 Processor -- start 2024-03-08T08:31:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.754+0000 7f23603e9640 1 -- start start 2024-03-08T08:31:30.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23580b2830 0x7f235815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 0x7f235815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358160750 con 0x7f235815f200 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580d07f0 con 0x7f2358160370 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f23603e9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235815f5e0 con 0x7f23580b2830 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39164/0 (socket says 172.21.15.12:39164) 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 -- 172.21.15.12:0/1561695474 learned_addr learned my addr 172.21.15.12:0/1561695474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235e986640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 0x7f235815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235e185640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23580b2830 0x7f235815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 -- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23580b2830 msgr2=0x7f235815dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23580b2830 0x7f235815dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 -- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 msgr2=0x7f235815ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 0x7f235815ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235d984640 1 -- 172.21.15.12:0/1561695474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580b5f60 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.755+0000 7f235e986640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 0x7f235815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f235d984640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f234c00e730 tx=0x7f234c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c00f040 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c004230 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23580b6240 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c00ee40 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.756+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580b6800 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.757+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580749c0 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.757+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f234c00c6d0 con 0x7f235815f200 2024-03-08T08:31:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.758+0000 7f2356ffd640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 0x7f234406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.758+0000 7f235e185640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 0x7f234406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.758+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f234c089f50 con 0x7f235815f200 2024-03-08T08:31:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.758+0000 7f235e185640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 0x7f234406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2348036000 tx=0x7f2348030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.762+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f234c017020 con 0x7f235815f200 2024-03-08T08:31:30.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.980+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23580cdd80 con 0x7f235815f200 2024-03-08T08:31:30.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.981+0000 7f2356ffd640 1 -- 172.21.15.12:0/1561695474 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f234c058700 con 0x7f235815f200 2024-03-08T08:31:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:30.982 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:31:30.983 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 msgr2=0x7f234406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 0x7f234406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2348036000 tx=0x7f2348030000 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 msgr2=0x7f23580b5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f234c00e730 tx=0x7f234c00ebf0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 shutdown_connections 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f234406b4d0 0x7f234406d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2358160370 0x7f235815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f235815f200 0x7f23580b5810 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 --2- 172.21.15.12:0/1561695474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23580b2830 0x7f235815dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 >> 172.21.15.12:0/1561695474 conn(0x7f23580b3460 msgr2=0x7f2358070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 shutdown_connections 2024-03-08T08:31:30.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:30.983+0000 7f23603e9640 1 -- 172.21.15.12:0/1561695474 wait complete. 2024-03-08T08:31:30.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.095+0000 7f3d4c377640 1 Processor -- start 2024-03-08T08:31:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.095+0000 7f3d4c377640 1 -- start start 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4c377640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3d440b9420 0x7f3d4415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4c377640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4415ebc0 con 0x7f3d4406c240 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4c377640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4415eea0 con 0x7f3d440d24b0 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4c377640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d440b6b30 con 0x7f3d440b9420 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4a113640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3d4406c240 0x7f3d4406c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34734/0 (socket says 172.21.15.12:34734) 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d4a113640 1 -- 172.21.15.12:0/714512907 learned_addr learned my addr 172.21.15.12:0/714512907 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49912640 1 --2- 172.21.15.12:0/714512907 >> v2:172.21.15.156:3300/0 conn(0x7f3d440b9420 0x7f3d4415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748454694 0 0) 0x7f3d4415ebc0 con 0x7f3d4406c240 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d2c003520 con 0x7f3d4406c240 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2739999098 0 0) 0x7f3d2c003520 con 0x7f3d4406c240 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 >> v2:172.21.15.156:3300/0 conn(0x7f3d440b9420 msgr2=0x7f3d4415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 --2- 172.21.15.12:0/714512907 >> v2:172.21.15.156:3300/0 conn(0x7f3d440b9420 0x7f3d4415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.096+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 >> v1:172.21.15.156:6789/0 conn(0x7f3d440d24b0 legacy=0x7f3d440d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440d1cb0 con 0x7f3d4406c240 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3d3c001670 con 0x7f3d4406c240 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 >> v1:172.21.15.12:6789/0 conn(0x7f3d4406c240 legacy=0x7f3d4406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c00c040 con 0x7f3d2c004800 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c00c1f0 con 0x7f3d2c0051c0 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c00c3f0 con 0x7f3d2c0039b0 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d4a914640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d4a113640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 -- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d2c0039b0 msgr2=0x7f3d2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 -- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d2c0051c0 msgr2=0x7f3d2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d49912640 1 -- 172.21.15.12:0/714512907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c003520 con 0x7f3d2c004800 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d4a914640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.097+0000 7f3d4a914640 1 -- 172.21.15.12:0/714512907 reap_dead start 2024-03-08T08:31:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d49912640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009a90 tx=0x7f3d2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4388ce50875b5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38039070 con 0x7f3d2c004800 2024-03-08T08:31:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3802c680 con 0x7f3d2c004800 2024-03-08T08:31:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d49111640 1 -- 172.21.15.12:0/714512907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38009690 con 0x7f3d2c004800 2024-03-08T08:31:31.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 -- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 msgr2=0x7f3d2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009a90 tx=0x7f3d2c005830 comp rx=0 tx=0).stop 2024-03-08T08:31:31.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 -- 172.21.15.12:0/714512907 shutdown_connections 2024-03-08T08:31:31.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 --2- 172.21.15.12:0/714512907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 -- 172.21.15.12:0/714512907 >> 172.21.15.12:0/714512907 conn(0x7f3d44072900 msgr2=0x7f3d440d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.098+0000 7f3d4c377640 1 -- 172.21.15.12:0/714512907 shutdown_connections 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.099+0000 7f3d4c377640 1 -- 172.21.15.12:0/714512907 wait complete. 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.100+0000 7f3d4c377640 1 Processor -- start 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.100+0000 7f3d4c377640 1 -- start start 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.100+0000 7f3d4c377640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.100+0000 7f3d4c377640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 0x7f3d440b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.100+0000 7f3d4c377640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d440b85f0 0x7f3d440cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4c377640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4415f180 con 0x7f3d2c004800 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4c377640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4415bc30 con 0x7f3d440b40b0 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4c377640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4406c620 con 0x7f3d440b85f0 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39188/0 (socket says 172.21.15.12:39188) 2024-03-08T08:31:31.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 -- 172.21.15.12:0/3854806529 learned_addr learned my addr 172.21.15.12:0/3854806529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d49912640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 0x7f3d440b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a914640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d440b85f0 0x7f3d440cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 -- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d440b85f0 msgr2=0x7f3d440cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d440b85f0 0x7f3d440cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 -- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 msgr2=0x7f3d440b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 0x7f3d440b80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 -- 172.21.15.12:0/3854806529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440cb840 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d49912640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 0x7f3d440b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4a113640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e7e0 tx=0x7f3d3c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00ee90 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c018070 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d440b35d0 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00cab0 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.101+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d440cb050 con 0x7f3d2c004800 2024-03-08T08:31:31.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.102+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d440749c0 con 0x7f3d2c004800 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.103+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d3c00cc50 con 0x7f3d2c004800 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.103+0000 7f3d42ffd640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 0x7f3d3006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.103+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3d3c08a060 con 0x7f3d2c004800 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.103+0000 7f3d49912640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 0x7f3d3006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.104+0000 7f3d49912640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 0x7f3d3006db30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d38036000 tx=0x7f3d38030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.108+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d3c058ef0 con 0x7f3d2c004800 2024-03-08T08:31:31.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.288+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d440b89d0 con 0x7f3d2c004800 2024-03-08T08:31:31.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.289+0000 7f3d42ffd640 1 -- 172.21.15.12:0/3854806529 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16859 (secure 0 0 0) 0x7f3d3c0587f0 con 0x7f3d2c004800 2024-03-08T08:31:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:31.291 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":37,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:55.107876+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.35","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:31.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 msgr2=0x7f3d3006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 0x7f3d3006db30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d38036000 tx=0x7f3d38030000 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 msgr2=0x7f3d44150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e7e0 tx=0x7f3d3c00eca0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 shutdown_connections 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3d3006b670 0x7f3d3006db30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3d440b85f0 0x7f3d440cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d440b40b0 0x7f3d440b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 --2- 172.21.15.12:0/3854806529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d2c004800 0x7f3d44150c10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 >> 172.21.15.12:0/3854806529 conn(0x7f3d44072900 msgr2=0x7f3d440ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 shutdown_connections 2024-03-08T08:31:31.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.291+0000 7f3d4c377640 1 -- 172.21.15.12:0/3854806529 wait complete. 2024-03-08T08:31:31.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T08:28:59.035221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T08:30:09.998602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T08:30:11.090210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T08:31:31.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T08:31:31.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.407+0000 7f94fc42a640 1 Processor -- start 2024-03-08T08:31:31.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.407+0000 7f94fc42a640 1 -- start start 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fc42a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d0250 0x7f94f40d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fc42a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f40d25d0 0x7f94f40d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fc42a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40b6be0 con 0x7f94f415dec0 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fc42a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40d2140 con 0x7f94f40d25d0 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fc42a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f415e2a0 con 0x7f94f40d0250 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f40d25d0 0x7f94f40d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f40d25d0 0x7f94f40d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39198/0 (socket says 172.21.15.12:39198) 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3826335907 learned_addr learned my addr 172.21.15.12:0/3826335907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94f99c5640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d0250 0x7f94f40d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3826335907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d0250 msgr2=0x7f94f40d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d0250 0x7f94f40d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3826335907 >> v1:172.21.15.156:6789/0 conn(0x7f94f415dec0 legacy=0x7f94f4161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.408+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3826335907 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f41620a0 con 0x7f94f40d25d0 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.409+0000 7f94fa9c7640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f40d25d0 0x7f94f40d2a60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009e50 tx=0x7f94e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bec1595e161b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.409+0000 7f94f91c4640 1 -- 172.21.15.12:0/3826335907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e402ca70 con 0x7f94f40d25d0 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.409+0000 7f94f91c4640 1 -- 172.21.15.12:0/3826335907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e402cc10 con 0x7f94f40d25d0 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.409+0000 7f94f91c4640 1 -- 172.21.15.12:0/3826335907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e40343f0 con 0x7f94f40d25d0 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 -- 172.21.15.12:0/3826335907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f40d25d0 msgr2=0x7f94f40d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f40d25d0 0x7f94f40d2a60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009e50 tx=0x7f94e402bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 -- 172.21.15.12:0/3826335907 shutdown_connections 2024-03-08T08:31:31.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f40d25d0 0x7f94f40d2a60 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3826335907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d0250 0x7f94f40d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 -- 172.21.15.12:0/3826335907 >> 172.21.15.12:0/3826335907 conn(0x7f94f406c240 msgr2=0x7f94f406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 -- 172.21.15.12:0/3826335907 shutdown_connections 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.410+0000 7f94fc42a640 1 -- 172.21.15.12:0/3826335907 wait complete. 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.411+0000 7f94fc42a640 1 Processor -- start 2024-03-08T08:31:31.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.411+0000 7f94fc42a640 1 -- start start 2024-03-08T08:31:31.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f40d0250 0x7f94f414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d25d0 0x7f94f415e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4162630 con 0x7f94f415dec0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4162350 con 0x7f94f40d0250 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fc42a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40d1dd0 con 0x7f94f40d25d0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39214/0 (socket says 172.21.15.12:39214) 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3981447968 learned_addr learned my addr 172.21.15.12:0/3981447968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94f99c5640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d25d0 0x7f94f415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa1c6640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f40d0250 0x7f94f414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d25d0 msgr2=0x7f94f415e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d25d0 0x7f94f415e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f40d0250 msgr2=0x7f94f414fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f40d0250 0x7f94f414fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 -- 172.21.15.12:0/3981447968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40cab80 con 0x7f94f415dec0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.412+0000 7f94fa9c7640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f94e4036040 tx=0x7f94e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4043020 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e403e450 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4009510 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f415c1e0 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f40cb100 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.413+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f40cb430 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.414+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94e4046020 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.414+0000 7f94f2ffd640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 0x7f94e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.414+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f94e40acb00 con 0x7f94f415dec0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.414+0000 7f94fa1c6640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 0x7f94e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.415+0000 7f94fa1c6640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 0x7f94e006d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f94ec006fd0 tx=0x7f94ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.419+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f94e404d030 con 0x7f94f415dec0 2024-03-08T08:31:31.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.597+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94f40749c0 con 0x7f94f415dec0 2024-03-08T08:31:31.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.598+0000 7f94f2ffd640 1 -- 172.21.15.12:0/3981447968 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94f40b6be0 con 0x7f94f415dec0 2024-03-08T08:31:31.599 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 msgr2=0x7f94e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 0x7f94e006d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f94ec006fd0 tx=0x7f94ec008040 comp rx=0 tx=0).stop 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 msgr2=0x7f94f415e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f94e4036040 tx=0x7f94e4004a40 comp rx=0 tx=0).stop 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 shutdown_connections 2024-03-08T08:31:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f94e006b4d0 0x7f94e006d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f415dec0 0x7f94f415e960 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f94f40d25d0 0x7f94f415e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 --2- 172.21.15.12:0/3981447968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94f40d0250 0x7f94f414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 >> 172.21.15.12:0/3981447968 conn(0x7f94f406c240 msgr2=0x7f94f40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 shutdown_connections 2024-03-08T08:31:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.600+0000 7f94fc42a640 1 -- 172.21.15.12:0/3981447968 wait complete. 2024-03-08T08:31:31.612 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T08:31:31.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.715+0000 7fbbe886b640 1 Processor -- start 2024-03-08T08:31:31.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.715+0000 7fbbe886b640 1 -- start start 2024-03-08T08:31:31.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe886b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbbe40b9760 0x7fbbe40d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe886b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbbe40d2050 0x7fbbe40b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe886b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe40b7250 con 0x7fbbe415dbc0 2024-03-08T08:31:31.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe886b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe40b7520 con 0x7fbbe40d2050 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe886b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe40b7800 con 0x7fbbe40b9760 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbbe40d2050 0x7fbbe40b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbbe40d2050 0x7fbbe40b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39218/0 (socket says 172.21.15.12:39218) 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/1616371436 learned_addr learned my addr 172.21.15.12:0/1616371436 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/1616371436 >> v2:172.21.15.156:3300/0 conn(0x7fbbe40b9760 msgr2=0x7fbbe40d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 --2- 172.21.15.12:0/1616371436 >> v2:172.21.15.156:3300/0 conn(0x7fbbe40b9760 0x7fbbe40d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/1616371436 >> v1:172.21.15.156:6789/0 conn(0x7fbbe415dbc0 legacy=0x7fbbe40b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.716+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/1616371436 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe40b7ae0 con 0x7fbbe40d2050 2024-03-08T08:31:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.717+0000 7fbbe2d9d640 1 --2- 172.21.15.12:0/1616371436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe40d2050 0x7fbbe40b85c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40cab30 tx=0x7fbbdc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79beddcddcda269 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.717+0000 7fbbe1d9b640 1 -- 172.21.15.12:0/1616371436 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc02cb20 con 0x7fbbe40d2050 2024-03-08T08:31:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.717+0000 7fbbe1d9b640 1 -- 172.21.15.12:0/1616371436 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc02ccc0 con 0x7fbbe40d2050 2024-03-08T08:31:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.717+0000 7fbbe1d9b640 1 -- 172.21.15.12:0/1616371436 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc034610 con 0x7fbbe40d2050 2024-03-08T08:31:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 -- 172.21.15.12:0/1616371436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe40d2050 msgr2=0x7fbbe40b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 --2- 172.21.15.12:0/1616371436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe40d2050 0x7fbbe40b85c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40cab30 tx=0x7fbbdc02bf80 comp rx=0 tx=0).stop 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 -- 172.21.15.12:0/1616371436 shutdown_connections 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 --2- 172.21.15.12:0/1616371436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe40d2050 0x7fbbe40b85c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 --2- 172.21.15.12:0/1616371436 >> v2:172.21.15.156:3300/0 conn(0x7fbbe40b9760 0x7fbbe40d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 -- 172.21.15.12:0/1616371436 >> 172.21.15.12:0/1616371436 conn(0x7fbbe4072720 msgr2=0x7fbbe4072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 -- 172.21.15.12:0/1616371436 shutdown_connections 2024-03-08T08:31:31.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.718+0000 7fbbe886b640 1 -- 172.21.15.12:0/1616371436 wait complete. 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.719+0000 7fbbe886b640 1 Processor -- start 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.719+0000 7fbbe886b640 1 -- start start 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbe40b9760 0x7fbbe415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 0x7fbbe41603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe415dfa0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe40b80a0 con 0x7fbbe40b9760 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe886b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe40b8b00 con 0x7fbbe40d2050 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39230/0 (socket says 172.21.15.12:39230) 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/754945082 learned_addr learned my addr 172.21.15.12:0/754945082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe259c640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbe40b9760 0x7fbbe415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbdbfff640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 0x7fbbe41603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 msgr2=0x7fbbe41603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 0x7fbbe41603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbe40b9760 msgr2=0x7fbbe415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbe40b9760 0x7fbbe415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 -- 172.21.15.12:0/754945082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe41608e0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbdbfff640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 0x7fbbe41603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.720+0000 7fbbe2d9d640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc036040 tx=0x7fbbdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc043020 con 0x7fbbe415dbc0 2024-03-08T08:31:31.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc03e450 con 0x7fbbe415dbc0 2024-03-08T08:31:31.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe4160b90 con 0x7fbbe415dbc0 2024-03-08T08:31:31.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc03e820 con 0x7fbbe415dbc0 2024-03-08T08:31:31.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe415e250 con 0x7fbbe415dbc0 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.721+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe4067ea0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.722+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbdc046020 con 0x7fbbe415dbc0 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.722+0000 7fbbdaffd640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.722+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbbdc0acbd0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.723+0000 7fbbe259c640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.723+0000 7fbbe259c640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 0x7fbbc406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4006fd0 tx=0x7fbbd4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:31.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.727+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbbe4067ea0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.907+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbbe40cf0d0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.908+0000 7fbbdaffd640 1 -- 172.21.15.12:0/754945082 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbbe40cf0d0 con 0x7fbbe415dbc0 2024-03-08T08:31:31.909 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 msgr2=0x7fbbc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 0x7fbbc406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4006fd0 tx=0x7fbbd4008040 comp rx=0 tx=0).stop 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 msgr2=0x7fbbe415ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc036040 tx=0x7fbbdc004a40 comp rx=0 tx=0).stop 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 shutdown_connections 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.909+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe415dbc0 0x7fbbe415ea50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbbe40d2050 0x7fbbe41603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 --2- 172.21.15.12:0/754945082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbe40b9760 0x7fbbe415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:31.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 >> 172.21.15.12:0/754945082 conn(0x7fbbe4072720 msgr2=0x7fbbe40ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:31.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 shutdown_connections 2024-03-08T08:31:31.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:31.910+0000 7fbbe886b640 1 -- 172.21.15.12:0/754945082 wait complete. 2024-03-08T08:31:31.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T08:31:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 Processor -- start 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 -- start start 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70dc15dec0 0x7f70dc161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f70dc161ed0 0x7f70dc0d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0b6be0 con 0x7f70dc0d0790 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc0d2140 con 0x7f70dc15dec0 2024-03-08T08:31:32.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.025+0000 7f70e1e9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc15e2a0 con 0x7f70dc161ed0 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70dc15dec0 0x7f70dc161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70dc15dec0 0x7f70dc161990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39244/0 (socket says 172.21.15.12:39244) 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 -- 172.21.15.12:0/498502159 learned_addr learned my addr 172.21.15.12:0/498502159 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70daffd640 1 --2- 172.21.15.12:0/498502159 >> v2:172.21.15.156:3300/0 conn(0x7f70dc161ed0 0x7f70dc0d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 -- 172.21.15.12:0/498502159 >> v2:172.21.15.156:3300/0 conn(0x7f70dc161ed0 msgr2=0x7f70dc0d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 --2- 172.21.15.12:0/498502159 >> v2:172.21.15.156:3300/0 conn(0x7f70dc161ed0 0x7f70dc0d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 -- 172.21.15.12:0/498502159 >> v1:172.21.15.156:6789/0 conn(0x7f70dc0d0790 legacy=0x7f70dc0b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.026+0000 7f70db7fe640 1 -- 172.21.15.12:0/498502159 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc162350 con 0x7f70dc15dec0 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70db7fe640 1 --2- 172.21.15.12:0/498502159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc15dec0 0x7f70dc161990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f70dc08ca20 tx=0x7f70cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9821bf3d3ea027df server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70da7fc640 1 -- 172.21.15.12:0/498502159 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc034070 con 0x7f70dc15dec0 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70da7fc640 1 -- 172.21.15.12:0/498502159 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc030860 con 0x7f70dc15dec0 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70da7fc640 1 -- 172.21.15.12:0/498502159 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc030c30 con 0x7f70dc15dec0 2024-03-08T08:31:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 -- 172.21.15.12:0/498502159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc15dec0 msgr2=0x7f70dc161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/498502159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc15dec0 0x7f70dc161990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f70dc08ca20 tx=0x7f70cc02b9a0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 -- 172.21.15.12:0/498502159 shutdown_connections 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/498502159 >> v2:172.21.15.156:3300/0 conn(0x7f70dc161ed0 0x7f70dc0d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/498502159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc15dec0 0x7f70dc161990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 -- 172.21.15.12:0/498502159 >> 172.21.15.12:0/498502159 conn(0x7f70dc06c240 msgr2=0x7f70dc06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 -- 172.21.15.12:0/498502159 shutdown_connections 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.027+0000 7f70e1e9a640 1 -- 172.21.15.12:0/498502159 wait complete. 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 Processor -- start 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 -- start start 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70dc0d0790 0x7f70dc0cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70dc15dec0 0x7f70dc0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0d0bd0 con 0x7f70dc161ed0 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0d1dd0 con 0x7f70dc0d0790 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.029+0000 7f70e1e9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0b94f0 con 0x7f70dc15dec0 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39248/0 (socket says 172.21.15.12:39248) 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 -- 172.21.15.12:0/361760474 learned_addr learned my addr 172.21.15.12:0/361760474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70db7fe640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70dc0d0790 0x7f70dc0cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70daffd640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70dc15dec0 0x7f70dc0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 -- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70dc15dec0 msgr2=0x7f70dc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70dc15dec0 0x7f70dc0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 -- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70dc0d0790 msgr2=0x7f70dc0cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70dc0d0790 0x7f70dc0cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 -- 172.21.15.12:0/361760474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0c75b0 con 0x7f70dc161ed0 2024-03-08T08:31:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70dbfff640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c730 tx=0x7f70d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003b70 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4003d30 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc0cc2a0 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4005310 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.030+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc0c7b50 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.031+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc0749c0 con 0x7f70dc161ed0 2024-03-08T08:31:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.032+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70d40056c0 con 0x7f70dc161ed0 2024-03-08T08:31:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.032+0000 7f70c3fff640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 0x7f70c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.032+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f70d408a140 con 0x7f70dc161ed0 2024-03-08T08:31:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.032+0000 7f70db7fe640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 0x7f70c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.032+0000 7f70db7fe640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 0x7f70c806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f70cc036000 tx=0x7f70cc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.036+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f70dc0b6be0 con 0x7f70dc161ed0 2024-03-08T08:31:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.216+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70dc0d1750 con 0x7f70dc161ed0 2024-03-08T08:31:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.216+0000 7f70c3fff640 1 -- 172.21.15.12:0/361760474 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70d4058980 con 0x7f70dc161ed0 2024-03-08T08:31:32.217 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 msgr2=0x7f70c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 0x7f70c806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f70cc036000 tx=0x7f70cc008030 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 msgr2=0x7f70dc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c730 tx=0x7f70d400cbf0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 shutdown_connections 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70c806b4d0 0x7f70c806d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70dc161ed0 0x7f70dc0cbd00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70dc15dec0 0x7f70dc0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 --2- 172.21.15.12:0/361760474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70dc0d0790 0x7f70dc0cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 >> 172.21.15.12:0/361760474 conn(0x7f70dc06c240 msgr2=0x7f70dc0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 shutdown_connections 2024-03-08T08:31:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.218+0000 7f70e1e9a640 1 -- 172.21.15.12:0/361760474 wait complete. 2024-03-08T08:31:32.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:31:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.331+0000 7fd4265f4640 1 Processor -- start 2024-03-08T08:31:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.331+0000 7fd4265f4640 1 -- start start 2024-03-08T08:31:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.331+0000 7fd4265f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd4200b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd4265f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42015df10 0x7fd420161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd4265f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420161fb0 con 0x7fd4200d0e10 2024-03-08T08:31:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd4265f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd420161d80 con 0x7fd42015df10 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd4265f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d1d60 con 0x7fd4200d0a30 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42015df10 0x7fd420161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42015df10 0x7fd420161840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39258/0 (socket says 172.21.15.12:39258) 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 -- 172.21.15.12:0/3990705008 learned_addr learned my addr 172.21.15.12:0/3990705008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41ffff640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd4200b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 -- 172.21.15.12:0/3990705008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 msgr2=0x7fd4200b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd4200b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 -- 172.21.15.12:0/3990705008 >> v1:172.21.15.156:6789/0 conn(0x7fd4200d0e10 legacy=0x7fd4200b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.332+0000 7fd41f7fe640 1 -- 172.21.15.12:0/3990705008 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420162290 con 0x7fd42015df10 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd41f7fe640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42015df10 0x7fd420161840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd414009870 tx=0x7fd41402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb234d37a8f47a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd41effd640 1 -- 172.21.15.12:0/3990705008 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414034070 con 0x7fd42015df10 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd41effd640 1 -- 172.21.15.12:0/3990705008 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414030840 con 0x7fd42015df10 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd41effd640 1 -- 172.21.15.12:0/3990705008 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414030c10 con 0x7fd42015df10 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd4265f4640 1 -- 172.21.15.12:0/3990705008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42015df10 msgr2=0x7fd420161840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd4265f4640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42015df10 0x7fd420161840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd414009870 tx=0x7fd41402b980 comp rx=0 tx=0).stop 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd4265f4640 1 -- 172.21.15.12:0/3990705008 shutdown_connections 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd4265f4640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42015df10 0x7fd420161840 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.333+0000 7fd4265f4640 1 --2- 172.21.15.12:0/3990705008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd4200b95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.334+0000 7fd4265f4640 1 -- 172.21.15.12:0/3990705008 >> 172.21.15.12:0/3990705008 conn(0x7fd42006c710 msgr2=0x7fd42006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.334+0000 7fd4265f4640 1 -- 172.21.15.12:0/3990705008 shutdown_connections 2024-03-08T08:31:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.334+0000 7fd4265f4640 1 -- 172.21.15.12:0/3990705008 wait complete. 2024-03-08T08:31:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.335+0000 7fd4265f4640 1 Processor -- start 2024-03-08T08:31:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.335+0000 7fd4265f4640 1 -- start start 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.335+0000 7fd4265f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd42015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd42015df10 0x7fd42015f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d12a0 con 0x7fd4200d0e10 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d1fb0 con 0x7fd42015df10 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d34e0 con 0x7fd4200d0a30 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39266/0 (socket says 172.21.15.12:39266) 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 -- 172.21.15.12:0/2289078836 learned_addr learned my addr 172.21.15.12:0/2289078836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41ffff640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd42015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd424b91640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd42015df10 0x7fd42015f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 -- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 msgr2=0x7fd42015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd42015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 -- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd42015df10 msgr2=0x7fd42015f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd42015df10 0x7fd42015f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 -- 172.21.15.12:0/2289078836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4201e7ad0 con 0x7fd4200d0e10 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41f7fe640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd41402b9c0 tx=0x7fd4140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414034040 con 0x7fd4200d0e10 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414039d90 con 0x7fd4200d0e10 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4201e7db0 con 0x7fd4200d0e10 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414043610 con 0x7fd4200d0e10 2024-03-08T08:31:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.336+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4201e8370 con 0x7fd4200d0e10 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.337+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4200749c0 con 0x7fd4200d0e10 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.338+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd41404a060 con 0x7fd4200d0e10 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.338+0000 7fd41cff9640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.338+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd4140ac280 con 0x7fd4200d0e10 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.338+0000 7fd41ffff640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.338+0000 7fd41ffff640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 0x7fd40c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd410006fd0 tx=0x7fd410008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.342+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd41403e070 con 0x7fd4200d0e10 2024-03-08T08:31:32.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.562+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4200d1630 con 0x7fd4200d0e10 2024-03-08T08:31:32.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.562+0000 7fd41cff9640 1 -- 172.21.15.12:0/2289078836 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7fd414037e40 con 0x7fd4200d0e10 2024-03-08T08:31:32.564 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:32.564 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:31:32.564 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 msgr2=0x7fd40c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 0x7fd40c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd410006fd0 tx=0x7fd410008040 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 msgr2=0x7fd4201f3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd41402b9c0 tx=0x7fd4140046c0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 shutdown_connections 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd42015df10 0x7fd42015f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4200d0e10 0x7fd4201f3cb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 --2- 172.21.15.12:0/2289078836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd4200d0a30 0x7fd42015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 >> 172.21.15.12:0/2289078836 conn(0x7fd42006c710 msgr2=0x7fd4200d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 shutdown_connections 2024-03-08T08:31:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.564+0000 7fd4265f4640 1 -- 172.21.15.12:0/2289078836 wait complete. 2024-03-08T08:31:32.576 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 Processor -- start 2024-03-08T08:31:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 -- start start 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 0x7fcc001688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0015e1a0 con 0x7fcc0015ddc0 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000b8790 con 0x7fcc000d1110 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcc0638c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000b9030 con 0x7fcc000b8300 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcbfffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcc0015ddc0 0x7fcc000d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34750/0 (socket says 172.21.15.12:34750) 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.677+0000 7fcbfffff640 1 -- 172.21.15.12:0/1553808315 learned_addr learned my addr 172.21.15.12:0/1553808315 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcc04929640 1 --2- 172.21.15.12:0/1553808315 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 0x7fcc001688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398554806 0 0) 0x7fcc0015e1a0 con 0x7fcc0015ddc0 2024-03-08T08:31:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe8003520 con 0x7fcc0015ddc0 2024-03-08T08:31:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684877586 0 0) 0x7fcc000b8790 con 0x7fcc000d1110 2024-03-08T08:31:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0015e1a0 con 0x7fcc000d1110 2024-03-08T08:31:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcc04929640 1 -- 172.21.15.12:0/1553808315 >> v1:172.21.15.156:6789/0 conn(0x7fcc000d1110 legacy=0x7fcc000b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcc04929640 1 -- 172.21.15.12:0/1553808315 >> v1:172.21.15.12:6789/0 conn(0x7fcc0015ddc0 legacy=0x7fcc000d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.678+0000 7fcc04929640 1 -- 172.21.15.12:0/1553808315 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000b8a70 con 0x7fcc000b8300 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.679+0000 7fcc04929640 1 --2- 172.21.15.12:0/1553808315 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 0x7fcc001688d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009dc0 tx=0x7fcbf802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cfddb9328bf0710 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.679+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8039070 con 0x7fcc000b8300 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.679+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf802ccd0 con 0x7fcc000b8300 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.679+0000 7fcbfeffd640 1 -- 172.21.15.12:0/1553808315 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8034660 con 0x7fcc000b8300 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 -- 172.21.15.12:0/1553808315 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 msgr2=0x7fcc001688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 --2- 172.21.15.12:0/1553808315 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 0x7fcc001688d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009dc0 tx=0x7fcbf802bf20 comp rx=0 tx=0).stop 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 -- 172.21.15.12:0/1553808315 shutdown_connections 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 --2- 172.21.15.12:0/1553808315 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc000b8300 0x7fcc001688d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 -- 172.21.15.12:0/1553808315 >> 172.21.15.12:0/1553808315 conn(0x7fcc00072720 msgr2=0x7fcc00072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 -- 172.21.15.12:0/1553808315 shutdown_connections 2024-03-08T08:31:32.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.680+0000 7fcc0638c640 1 -- 172.21.15.12:0/1553808315 wait complete. 2024-03-08T08:31:32.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.681+0000 7fcc0638c640 1 Processor -- start 2024-03-08T08:31:32.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.681+0000 7fcc0638c640 1 -- start start 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 0x7fcc000ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc00151260 0x7fcc001f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000b9980 con 0x7fcc0015ddc0 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000d1d20 con 0x7fcc000d1110 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000d31c0 con 0x7fcc00151260 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39276/0 (socket says 172.21.15.12:39276) 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 -- 172.21.15.12:0/4190309151 learned_addr learned my addr 172.21.15.12:0/4190309151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbfffff640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 0x7fcc000ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc04929640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc00151260 0x7fcc001f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 -- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc00151260 msgr2=0x7fcc001f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc00151260 0x7fcc001f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 -- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 msgr2=0x7fcc000ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 0x7fcc000ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 -- 172.21.15.12:0/4190309151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0015a500 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbfffff640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 0x7fcc000ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbff7fe640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c500 tx=0x7fcbf400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40052d0 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4005490 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0015ff80 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4012870 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.682+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc00160540 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.683+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc000749c0 con 0x7fcc0015ddc0 2024-03-08T08:31:32.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.684+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf4012a10 con 0x7fcc0015ddc0 2024-03-08T08:31:32.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.684+0000 7fcbfcff9640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.684+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcbf4089f70 con 0x7fcc0015ddc0 2024-03-08T08:31:32.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.684+0000 7fcbfffff640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.685+0000 7fcbfffff640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 0x7fcbec06d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00011e0 tx=0x7fcbf0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.688+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcbf4058d80 con 0x7fcc0015ddc0 2024-03-08T08:31:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.872+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc000b7410 con 0x7fcc0015ddc0 2024-03-08T08:31:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.873+0000 7fcbfcff9640 1 -- 172.21.15.12:0/4190309151 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16859 (secure 0 0 0) 0x7fcbf4058680 con 0x7fcc0015ddc0 2024-03-08T08:31:32.874 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:32.875 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":37,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:55.107876+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.35","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 msgr2=0x7fcbec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 0x7fcbec06d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00011e0 tx=0x7fcbf0009040 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 msgr2=0x7fcc00150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c500 tx=0x7fcbf400c9c0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 shutdown_connections 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcc00151260 0x7fcc001f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc0015ddc0 0x7fcc00150d20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 --2- 172.21.15.12:0/4190309151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc000d1110 0x7fcc000ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 >> 172.21.15.12:0/4190309151 conn(0x7fcc00072720 msgr2=0x7fcc00072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 shutdown_connections 2024-03-08T08:31:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.875+0000 7fcc0638c640 1 -- 172.21.15.12:0/4190309151 wait complete. 2024-03-08T08:31:32.887 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-08T08:31:32.888 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:31:32.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.988+0000 7f57ac288640 1 Processor -- start 2024-03-08T08:31:32.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.988+0000 7f57ac288640 1 -- start start 2024-03-08T08:31:32.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57ac288640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 0x7f57a4161820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57ac288640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57ac288640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a406dd00 con 0x7f57a40b2df0 2024-03-08T08:31:32.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57ac288640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a406eda0 con 0x7f57a415dc20 2024-03-08T08:31:32.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57ac288640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406dab0 con 0x7f57a4161d60 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57a9823640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57aa024640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 0x7f57a4161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57aa825640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f57a40b2df0 0x7f57a40b3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:35328/0 (socket says 172.21.15.12:35328) 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57a9823640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:40952/0 (socket says 172.21.15.12:40952) 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.989+0000 7f57aa825640 1 -- 172.21.15.12:0/3510690910 learned_addr learned my addr 172.21.15.12:0/3510690910 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.990+0000 7f57a9823640 1 -- 172.21.15.12:0/3510690910 >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 msgr2=0x7f57a4161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.990+0000 7f57a9823640 1 --2- 172.21.15.12:0/3510690910 >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 0x7f57a4161820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.990+0000 7f57a9823640 1 -- 172.21.15.12:0/3510690910 >> v1:172.21.15.156:6789/0 conn(0x7f57a40b2df0 legacy=0x7f57a40b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.990+0000 7f57a9823640 1 -- 172.21.15.12:0/3510690910 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a406dfe0 con 0x7f57a4161d60 2024-03-08T08:31:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.990+0000 7f57aa024640 1 --2- 172.21.15.12:0/3510690910 >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 0x7f57a4161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57a9823640 1 --2- 172.21.15.12:0/3510690910 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57940097a0 tx=0x7f579402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfaaf04fe06538eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57a9022640 1 -- 172.21.15.12:0/3510690910 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57940315a0 con 0x7f57a4161d60 2024-03-08T08:31:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57a9022640 1 -- 172.21.15.12:0/3510690910 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794037070 con 0x7f57a4161d60 2024-03-08T08:31:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57a9022640 1 -- 172.21.15.12:0/3510690910 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794031960 con 0x7f57a4161d60 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 -- 172.21.15.12:0/3510690910 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 msgr2=0x7f57a41621c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 --2- 172.21.15.12:0/3510690910 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57940097a0 tx=0x7f579402b840 comp rx=0 tx=0).stop 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 -- 172.21.15.12:0/3510690910 shutdown_connections 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 --2- 172.21.15.12:0/3510690910 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a4161d60 0x7f57a41621c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 --2- 172.21.15.12:0/3510690910 >> v2:172.21.15.156:3300/0 conn(0x7f57a415dc20 0x7f57a4161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 -- 172.21.15.12:0/3510690910 >> 172.21.15.12:0/3510690910 conn(0x7f57a40b1af0 msgr2=0x7f57a40b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 -- 172.21.15.12:0/3510690910 shutdown_connections 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.991+0000 7f57ac288640 1 -- 172.21.15.12:0/3510690910 wait complete. 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.992+0000 7f57ac288640 1 Processor -- start 2024-03-08T08:31:32.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 -- start start 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a40b2df0 0x7f57a415fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a415dc20 0x7f57a414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406e5a0 con 0x7f57a4161d60 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406e910 con 0x7f57a415dc20 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57ac288640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a40b50c0 con 0x7f57a40b2df0 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57aa825640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57aa825640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39284/0 (socket says 172.21.15.12:39284) 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57aa825640 1 -- 172.21.15.12:0/4007345912 learned_addr learned my addr 172.21.15.12:0/4007345912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57aa024640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a40b2df0 0x7f57a415fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.993+0000 7f57a9823640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a415dc20 0x7f57a414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 -- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a40b2df0 msgr2=0x7f57a415fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a40b2df0 0x7f57a415fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 -- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a415dc20 msgr2=0x7f57a414e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a415dc20 0x7f57a414e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 -- 172.21.15.12:0/4007345912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4160320 con 0x7f57a4161d60 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57aa825640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f57a000ca80 tx=0x7f57a000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a0017070 con 0x7f57a4161d60 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a000ac30 con 0x7f57a4161d60 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a41605c0 con 0x7f57a4161d60 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a0012600 con 0x7f57a4161d60 2024-03-08T08:31:32.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.994+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a4160b80 con 0x7f57a4161d60 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.995+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a415e0b0 con 0x7f57a4161d60 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.995+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57a001a070 con 0x7f57a4161d60 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.996+0000 7f579affd640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 0x7f578806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.996+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f57a0089ec0 con 0x7f57a4161d60 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.996+0000 7f57aa024640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 0x7f578806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:32.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:32.996+0000 7f57aa024640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 0x7f578806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f579c009560 tx=0x7f579c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.000+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f57a000ae20 con 0x7f57a4161d60 2024-03-08T08:31:33.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.218+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57a40749c0 con 0x7f57a4161d60 2024-03-08T08:31:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.219+0000 7f579affd640 1 -- 172.21.15.12:0/4007345912 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f57a0058670 con 0x7f57a4161d60 2024-03-08T08:31:33.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:33.220 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:31:33.220 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 msgr2=0x7f578806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 0x7f578806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f579c009560 tx=0x7f579c009190 comp rx=0 tx=0).stop 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 msgr2=0x7f57a40af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f57a000ca80 tx=0x7f57a000cf40 comp rx=0 tx=0).stop 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 shutdown_connections 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f578806b4d0 0x7f578806d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57a4161d60 0x7f57a40af120 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a415dc20 0x7f57a414e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 --2- 172.21.15.12:0/4007345912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57a40b2df0 0x7f57a415fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 >> 172.21.15.12:0/4007345912 conn(0x7f57a40b1af0 msgr2=0x7f57a4159390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 shutdown_connections 2024-03-08T08:31:33.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.221+0000 7f57ac288640 1 -- 172.21.15.12:0/4007345912 wait complete. 2024-03-08T08:31:33.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:31:33.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.332+0000 7efc61d0c640 1 Processor -- start 2024-03-08T08:31:33.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.332+0000 7efc61d0c640 1 -- start start 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc61d0c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc5c0d0a30 0x7efc5c0b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc61d0c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efc5c15df10 0x7efc5c161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc61d0c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5c161fb0 con 0x7efc5c0d0e10 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc61d0c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5c161d80 con 0x7efc5c0d0a30 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc61d0c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5c0d1d60 con 0x7efc5c15df10 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc5c0d0a30 0x7efc5c0b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efc5c0d0e10 0x7efc5c0b54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34752/0 (socket says 172.21.15.12:34752) 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc5c0d0a30 0x7efc5c0b95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39290/0 (socket says 172.21.15.12:39290) 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5bfff640 1 -- 172.21.15.12:0/2358201318 learned_addr learned my addr 172.21.15.12:0/2358201318 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:33.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5affd640 1 --2- 172.21.15.12:0/2358201318 >> v2:172.21.15.156:3300/0 conn(0x7efc5c15df10 0x7efc5c161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 -- 172.21.15.12:0/2358201318 >> v2:172.21.15.156:3300/0 conn(0x7efc5c15df10 msgr2=0x7efc5c161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/2358201318 >> v2:172.21.15.156:3300/0 conn(0x7efc5c15df10 0x7efc5c161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 -- 172.21.15.12:0/2358201318 >> v1:172.21.15.12:6789/0 conn(0x7efc5c0d0e10 legacy=0x7efc5c0b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5b7fe640 1 -- 172.21.15.12:0/2358201318 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c162290 con 0x7efc5c0d0a30 2024-03-08T08:31:33.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.333+0000 7efc5a7fc640 1 -- 172.21.15.12:0/2358201318 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429912044 0 0) 0x7efc5c161fb0 con 0x7efc5c0d0e10 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.334+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/2358201318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c0d0a30 0x7efc5c0b95a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0d4ae0 tx=0x7efc4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2141c36143fdc273 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.334+0000 7efc5a7fc640 1 -- 172.21.15.12:0/2358201318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c035020 con 0x7efc5c0d0a30 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.334+0000 7efc5a7fc640 1 -- 172.21.15.12:0/2358201318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c031830 con 0x7efc5c0d0a30 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.334+0000 7efc5a7fc640 1 -- 172.21.15.12:0/2358201318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c031c00 con 0x7efc5c0d0a30 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 -- 172.21.15.12:0/2358201318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c0d0a30 msgr2=0x7efc5c0b95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 --2- 172.21.15.12:0/2358201318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c0d0a30 0x7efc5c0b95a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0d4ae0 tx=0x7efc4c02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 -- 172.21.15.12:0/2358201318 shutdown_connections 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 --2- 172.21.15.12:0/2358201318 >> v2:172.21.15.156:3300/0 conn(0x7efc5c15df10 0x7efc5c161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 --2- 172.21.15.12:0/2358201318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c0d0a30 0x7efc5c0b95a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 -- 172.21.15.12:0/2358201318 >> 172.21.15.12:0/2358201318 conn(0x7efc5c06c710 msgr2=0x7efc5c06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 -- 172.21.15.12:0/2358201318 shutdown_connections 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.335+0000 7efc61d0c640 1 -- 172.21.15.12:0/2358201318 wait complete. 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.336+0000 7efc61d0c640 1 Processor -- start 2024-03-08T08:31:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.336+0000 7efc61d0c640 1 -- start start 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 0x7efc5c1e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc5c0d0e10 0x7efc5c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0d1260 con 0x7efc5c15df10 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0d1fb0 con 0x7efc5c0d0e10 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc61d0c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0d13e0 con 0x7efc5c0d0a30 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39302/0 (socket says 172.21.15.12:39302) 2024-03-08T08:31:33.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 -- 172.21.15.12:0/3982067452 learned_addr learned my addr 172.21.15.12:0/3982067452 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5affd640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc5c0d0e10 0x7efc5c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 0x7efc5c1e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 -- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 msgr2=0x7efc5c1e75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 0x7efc5c1e75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 -- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc5c0d0e10 msgr2=0x7efc5c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc5c0d0e10 0x7efc5c158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5bfff640 1 -- 172.21.15.12:0/3982067452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c15fac0 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.337+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 0x7efc5c1e75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc5bfff640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efc5400c7d0 tx=0x7efc5400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54018070 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5400a640 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c15fd60 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5400aa70 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.338+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c1e8ba0 con 0x7efc5c15df10 2024-03-08T08:31:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.339+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c0749c0 con 0x7efc5c15df10 2024-03-08T08:31:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.339+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc5401b070 con 0x7efc5c15df10 2024-03-08T08:31:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.340+0000 7efc43fff640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.340+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efc540897d0 con 0x7efc5c15df10 2024-03-08T08:31:33.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.340+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.340+0000 7efc5b7fe640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 0x7efc4806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc4c036000 tx=0x7efc4c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.344+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efc54058660 con 0x7efc5c15df10 2024-03-08T08:31:33.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.562+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc5c0d1630 con 0x7efc5c15df10 2024-03-08T08:31:33.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.562+0000 7efc43fff640 1 -- 172.21.15.12:0/3982067452 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4943 (secure 0 0 0) 0x7efc54057f60 con 0x7efc5c15df10 2024-03-08T08:31:33.563 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:33.564 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T08:30:12.121225+0000","modified":"2024-03-08T08:30:14.788469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5277":{"gid":5277,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6837/2820296658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2820296658},{"type":"v1","addr":"172.21.15.12:6837","nonce":2820296658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-08T08:31:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-08T08:31:33.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 msgr2=0x7efc4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 0x7efc4806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc4c036000 tx=0x7efc4c008030 comp rx=0 tx=0).stop 2024-03-08T08:31:33.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 msgr2=0x7efc5c1f3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efc5400c7d0 tx=0x7efc5400cc90 comp rx=0 tx=0).stop 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 shutdown_connections 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc5c15df10 0x7efc5c1f3cc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc5c0d0e10 0x7efc5c158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 --2- 172.21.15.12:0/3982067452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efc5c0d0a30 0x7efc5c1e75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 >> 172.21.15.12:0/3982067452 conn(0x7efc5c06c710 msgr2=0x7efc5c0b8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 shutdown_connections 2024-03-08T08:31:33.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.564+0000 7efc61d0c640 1 -- 172.21.15.12:0/3982067452 wait complete. 2024-03-08T08:31:33.576 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:33.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.676+0000 7f5c8e8eb640 1 Processor -- start 2024-03-08T08:31:33.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.676+0000 7f5c8e8eb640 1 -- start start 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8e8eb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5c8806c240 0x7f5c8806c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8e8eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c8815e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8e8eb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8806cb60 con 0x7f5c880b5550 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8e8eb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c880b6b30 con 0x7f5c8806c240 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8e8eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8815ebc0 con 0x7f5c880b9420 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8ce88640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5c880b5550 0x7f5c880b5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34760/0 (socket says 172.21.15.12:34760) 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c8ce88640 1 -- 172.21.15.12:0/27619309 learned_addr learned my addr 172.21.15.12:0/27619309 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409434223 0 0) 0x7f5c8806cb60 con 0x7f5c880b5550 2024-03-08T08:31:33.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c87fff640 1 --2- 172.21.15.12:0/27619309 >> v2:172.21.15.156:3300/0 conn(0x7f5c8806c240 0x7f5c8806c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c70003520 con 0x7f5c880b5550 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.677+0000 7f5c877fe640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c8815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2571519411 0 0) 0x7f5c70003520 con 0x7f5c880b5550 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 msgr2=0x7f5c8815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c8815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 >> v2:172.21.15.156:3300/0 conn(0x7f5c8806c240 msgr2=0x7f5c8806c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 --2- 172.21.15.12:0/27619309 >> v2:172.21.15.156:3300/0 conn(0x7f5c8806c240 0x7f5c8806c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8815eea0 con 0x7f5c880b5550 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f5c800019f0 con 0x7f5c880b5550 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 >> v1:172.21.15.12:6789/0 conn(0x7f5c880b5550 legacy=0x7f5c880b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c700039b0 0x7f5c70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8815ebc0 con 0x7f5c70004d40 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7000b9c0 con 0x7f5c700039b0 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7000bb40 con 0x7f5c880b9420 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c87fff640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c700039b0 0x7f5c70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c877fe640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 -- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 msgr2=0x7f5c70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880b9420 0x7f5c70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 -- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c700039b0 msgr2=0x7f5c70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c700039b0 0x7f5c70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.678+0000 7f5c8ce88640 1 -- 172.21.15.12:0/27619309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70003520 con 0x7f5c70004d40 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.679+0000 7f5c8ce88640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5c800013d0 tx=0x7f5c70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42e7717dc627af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.679+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000a4e0 con 0x7f5c70004d40 2024-03-08T08:31:33.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.679+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c80034650 con 0x7f5c70004d40 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.679+0000 7f5c86ffd640 1 -- 172.21.15.12:0/27619309 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80034a40 con 0x7f5c70004d40 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/27619309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c70004d40 msgr2=0x7f5c700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/27619309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5c800013d0 tx=0x7f5c70005830 comp rx=0 tx=0).stop 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8ce88640 1 -- 172.21.15.12:0/27619309 reap_dead start 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/27619309 shutdown_connections 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/27619309 >> 172.21.15.12:0/27619309 conn(0x7f5c88072900 msgr2=0x7f5c880d3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/27619309 shutdown_connections 2024-03-08T08:31:33.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.680+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/27619309 wait complete. 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.681+0000 7f5c8e8eb640 1 Processor -- start 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.681+0000 7f5c8e8eb640 1 -- start start 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880ccf20 0x7f5c880b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c88156a70 0x7f5c88156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8815bc30 con 0x7f5c880b87d0 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880d1f90 con 0x7f5c88156a70 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8815f180 con 0x7f5c880ccf20 2024-03-08T08:31:33.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39316/0 (socket says 172.21.15.12:39316) 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 -- 172.21.15.12:0/258006974 learned_addr learned my addr 172.21.15.12:0/258006974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 -- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880ccf20 msgr2=0x7f5c880b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880ccf20 0x7f5c880b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 -- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c88156a70 msgr2=0x7f5c88156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c88156a70 0x7f5c88156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 -- 172.21.15.12:0/258006974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880b6e80 con 0x7f5c880b87d0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c877fe640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c011940 tx=0x7f5c7c011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c019070 con 0x7f5c880b87d0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c004240 con 0x7f5c880b87d0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c880b7160 con 0x7f5c880b87d0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.682+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00f860 con 0x7f5c880b87d0 2024-03-08T08:31:33.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.683+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c880b7720 con 0x7f5c880b87d0 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.684+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c880749c0 con 0x7f5c880b87d0 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.684+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c7c01c070 con 0x7f5c880b87d0 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.684+0000 7f5c84ff9640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 0x7f5c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.684+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5c7c08a6d0 con 0x7f5c880b87d0 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.684+0000 7f5c87fff640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 0x7f5c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.685+0000 7f5c87fff640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 0x7f5c7406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c78006b10 tx=0x7f5c78006aa0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.689+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5c7c0594d0 con 0x7f5c880b87d0 2024-03-08T08:31:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.870+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c880b3210 con 0x7f5c880b87d0 2024-03-08T08:31:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.871+0000 7f5c84ff9640 1 -- 172.21.15.12:0/258006974 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16859 (secure 0 0 0) 0x7f5c7c058dd0 con 0x7f5c880b87d0 2024-03-08T08:31:33.872 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:33.873 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":37,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:30:55.107876+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-08T08:30:09.998602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-08T08:30:11.090210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.35","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 msgr2=0x7f5c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 0x7f5c7406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c78006b10 tx=0x7f5c78006aa0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 msgr2=0x7f5c880b8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c011940 tx=0x7f5c7c011e00 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 shutdown_connections 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5c7406b4d0 0x7f5c7406d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c88156a70 0x7f5c88156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c880b87d0 0x7f5c880b8c50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 --2- 172.21.15.12:0/258006974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c880ccf20 0x7f5c880b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 >> 172.21.15.12:0/258006974 conn(0x7f5c88072900 msgr2=0x7f5c880b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 shutdown_connections 2024-03-08T08:31:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.873+0000 7f5c8e8eb640 1 -- 172.21.15.12:0/258006974 wait complete. 2024-03-08T08:31:33.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-08T08:31:33.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.986+0000 7f994d6fb640 1 Processor -- start 2024-03-08T08:31:33.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.986+0000 7f994d6fb640 1 -- start start 2024-03-08T08:31:33.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f994d6fb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f994815de20 0x7f99480d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f994d6fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f99480b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f994d6fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480d26c0 con 0x7f99480b94a0 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f994d6fb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99480d1920 con 0x7f994815de20 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f994d6fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480d32d0 con 0x7f994806c920 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f994815de20 0x7f99480d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f994815de20 0x7f99480d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39318/0 (socket says 172.21.15.12:39318) 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 -- 172.21.15.12:0/809541416 learned_addr learned my addr 172.21.15.12:0/809541416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 -- 172.21.15.12:0/809541416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 msgr2=0x7f99480b6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 --2- 172.21.15.12:0/809541416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f99480b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 -- 172.21.15.12:0/809541416 >> v1:172.21.15.156:6789/0 conn(0x7f99480b94a0 legacy=0x7f994806c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:33.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.987+0000 7f9946ffd640 1 -- 172.21.15.12:0/809541416 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480d29a0 con 0x7f994815de20 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.988+0000 7f9946ffd640 1 --2- 172.21.15.12:0/809541416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f99480d1fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9940009870 tx=0x7f994002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654e717d6cc1b6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.988+0000 7f9945ffb640 1 -- 172.21.15.12:0/809541416 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940035020 con 0x7f994815de20 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.988+0000 7f9945ffb640 1 -- 172.21.15.12:0/809541416 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940031640 con 0x7f994815de20 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.988+0000 7f9945ffb640 1 -- 172.21.15.12:0/809541416 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994002d040 con 0x7f994815de20 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 -- 172.21.15.12:0/809541416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 msgr2=0x7f99480d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 --2- 172.21.15.12:0/809541416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f99480d1fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9940009870 tx=0x7f994002b8f0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 -- 172.21.15.12:0/809541416 shutdown_connections 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 --2- 172.21.15.12:0/809541416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f99480b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 --2- 172.21.15.12:0/809541416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f99480d1fc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 -- 172.21.15.12:0/809541416 >> 172.21.15.12:0/809541416 conn(0x7f9948072930 msgr2=0x7f99480d3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:33.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 -- 172.21.15.12:0/809541416 shutdown_connections 2024-03-08T08:31:33.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.989+0000 7f994d6fb640 1 -- 172.21.15.12:0/809541416 wait complete. 2024-03-08T08:31:33.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.990+0000 7f994d6fb640 1 Processor -- start 2024-03-08T08:31:33.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.990+0000 7f994d6fb640 1 -- start start 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f994814e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 0x7f99480b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480d3760 con 0x7f994815de20 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480d2f60 con 0x7f99480b94a0 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480d38e0 con 0x7f994806c920 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39328/0 (socket says 172.21.15.12:39328) 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 -- 172.21.15.12:0/2052334636 learned_addr learned my addr 172.21.15.12:0/2052334636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f9946ffd640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f994814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99467fc640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 0x7f99480b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 -- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 msgr2=0x7f994814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f994814e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 -- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 msgr2=0x7f99480b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 0x7f99480b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 -- 172.21.15.12:0/2052334636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994814ee40 con 0x7f994815de20 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99467fc640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 0x7f99480b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f99477fe640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f993800b970 tx=0x7f993800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938019070 con 0x7f994815de20 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800cb60 con 0x7f994815de20 2024-03-08T08:31:33.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938014440 con 0x7f994815de20 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99480c86f0 con 0x7f994815de20 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.991+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480c8cb0 con 0x7f994815de20 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.992+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99480749c0 con 0x7f994815de20 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.993+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993801c070 con 0x7f994815de20 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.993+0000 7f993f7fe640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 0x7f992806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.993+0000 7f9946ffd640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 0x7f992806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:33.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.993+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9938089e80 con 0x7f994815de20 2024-03-08T08:31:33.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.994+0000 7f9946ffd640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 0x7f992806d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9940036000 tx=0x7f994003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:33.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:33.998+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f99380586c0 con 0x7f994815de20 2024-03-08T08:31:34.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.218+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f994815bc30 con 0x7f994815de20 2024-03-08T08:31:34.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-08T08:31:34.219+0000 7f49b9ba2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T08:31:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.220+0000 7f993f7fe640 1 -- 172.21.15.12:0/2052334636 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f993805c760 con 0x7f994815de20 2024-03-08T08:31:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.223+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 msgr2=0x7f992806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.223+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 0x7f992806d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9940036000 tx=0x7f994003c040 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.223+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 msgr2=0x7f9948153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.223+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f993800b970 tx=0x7f993800be30 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 shutdown_connections 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f992806b4d0 0x7f992806d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f994815de20 0x7f9948153d60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99480b94a0 0x7f99480b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 --2- 172.21.15.12:0/2052334636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994806c920 0x7f994814e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 >> 172.21.15.12:0/2052334636 conn(0x7f9948072930 msgr2=0x7f99481511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:34.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 shutdown_connections 2024-03-08T08:31:34.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.224+0000 7f994d6fb640 1 -- 172.21.15.12:0/2052334636 wait complete. 2024-03-08T08:31:34.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-08T08:31:34.261 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-08T08:31:34.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.339+0000 7f2419a4e640 1 Processor -- start 2024-03-08T08:31:34.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 -- start start 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 0x7f24140cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f241406bef0 0x7f24140cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140d0290 con 0x7f241415e0d0 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140d0570 con 0x7f241406bef0 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f2419a4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140d0850 con 0x7f2414099b90 2024-03-08T08:31:34.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f24137fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f241406bef0 0x7f24140cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f24127fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f241415e0d0 0x7f2414099f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34774/0 (socket says 172.21.15.12:34774) 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.340+0000 7f24127fc640 1 -- 172.21.15.12:0/745888052 learned_addr learned my addr 172.21.15.12:0/745888052 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f2412ffd640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 0x7f24140cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f24137fe640 1 -- 172.21.15.12:0/745888052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 msgr2=0x7f24140cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f24137fe640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 0x7f24140cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f24137fe640 1 -- 172.21.15.12:0/745888052 >> v1:172.21.15.12:6789/0 conn(0x7f241415e0d0 legacy=0x7f2414099f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f24137fe640 1 -- 172.21.15.12:0/745888052 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140d0b30 con 0x7f241406bef0 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f2411ffb640 1 -- 172.21.15.12:0/745888052 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043947075 0 0) 0x7f24140d0290 con 0x7f241415e0d0 2024-03-08T08:31:34.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f2412ffd640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 0x7f24140cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.341+0000 7f24137fe640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140cbfc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f240c0097a0 tx=0x7f240c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17acf21e57e3f241 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2411ffb640 1 -- 172.21.15.12:0/745888052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c035020 con 0x7f241406bef0 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2411ffb640 1 -- 172.21.15.12:0/745888052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c031830 con 0x7f241406bef0 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2411ffb640 1 -- 172.21.15.12:0/745888052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c031c00 con 0x7f241406bef0 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 -- 172.21.15.12:0/745888052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 msgr2=0x7f24140cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140cbfc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f240c0097a0 tx=0x7f240c02b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 -- 172.21.15.12:0/745888052 shutdown_connections 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140cbfc0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 --2- 172.21.15.12:0/745888052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2414099b90 0x7f24140cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 -- 172.21.15.12:0/745888052 >> 172.21.15.12:0/745888052 conn(0x7f24140b3460 msgr2=0x7f24140b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 -- 172.21.15.12:0/745888052 shutdown_connections 2024-03-08T08:31:34.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.342+0000 7f2419a4e640 1 -- 172.21.15.12:0/745888052 wait complete. 2024-03-08T08:31:34.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.343+0000 7f2419a4e640 1 Processor -- start 2024-03-08T08:31:34.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 -- start start 2024-03-08T08:31:34.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2414099b90 0x7f2414158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24141f6dc0 0x7f241415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241409b310 con 0x7f241406bef0 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241409b190 con 0x7f2414099b90 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2419a4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140d1330 con 0x7f24141f6dc0 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2412ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2412ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39362/0 (socket says 172.21.15.12:39362) 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.344+0000 7f2412ffd640 1 -- 172.21.15.12:0/2784926538 learned_addr learned my addr 172.21.15.12:0/2784926538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f24137fe640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24141f6dc0 0x7f241415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f24127fc640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2414099b90 0x7f2414158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 -- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24141f6dc0 msgr2=0x7f241415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24141f6dc0 0x7f241415ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 -- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2414099b90 msgr2=0x7f2414158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:34.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2414099b90 0x7f2414158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 -- 172.21.15.12:0/2784926538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24141f72c0 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2412ffd640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f240400cf60 tx=0x7f240400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240401a070 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240400d840 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24141f75a0 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240400dc10 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.345+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24141f7b60 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.346+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24140749c0 con 0x7f241406bef0 2024-03-08T08:31:34.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.346+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240401d070 con 0x7f241406bef0 2024-03-08T08:31:34.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.347+0000 7f23fbfff640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 0x7f240006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:34.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.347+0000 7f24127fc640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 0x7f240006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:34.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.347+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f240408aab0 con 0x7f241406bef0 2024-03-08T08:31:34.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.347+0000 7f24127fc640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 0x7f240006d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f24140b04a0 tx=0x7f2408007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:34.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.351+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2404059210 con 0x7f241406bef0 2024-03-08T08:31:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:34.576+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f24140b78f0 con 0x7f241406bef0 2024-03-08T08:31:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.224+0000 7f23fbfff640 1 -- 172.21.15.12:0/2784926538 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f240405d2b0 con 0x7f241406bef0 2024-03-08T08:31:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 msgr2=0x7f240006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 0x7f240006d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f24140b04a0 tx=0x7f2408007c00 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 msgr2=0x7f24140b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f240400cf60 tx=0x7f240400cf90 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 shutdown_connections 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f240006b4d0 0x7f240006d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24141f6dc0 0x7f241415ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2414099b90 0x7f2414158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 --2- 172.21.15.12:0/2784926538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f241406bef0 0x7f24140b66e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 >> 172.21.15.12:0/2784926538 conn(0x7f24140b3460 msgr2=0x7f24140cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 shutdown_connections 2024-03-08T08:31:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.227+0000 7f2419a4e640 1 -- 172.21.15.12:0/2784926538 wait complete. 2024-03-08T08:31:35.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-08T08:31:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.395+0000 7fac2ac15640 1 Processor -- start 2024-03-08T08:31:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.395+0000 7fac2ac15640 1 -- start start 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac2ac15640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fac2415df90 0x7fac24161950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac2ac15640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac240b9600 0x7fac240ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac2ac15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac24162050 con 0x7fac240cedf0 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac2ac15640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac240cf240 con 0x7fac240b9600 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac2ac15640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac240d2070 con 0x7fac2415df90 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac23fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac240b9600 0x7fac240ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac23fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac240b9600 0x7fac240ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39368/0 (socket says 172.21.15.12:39368) 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.396+0000 7fac23fff640 1 -- 172.21.15.12:0/2820762171 learned_addr learned my addr 172.21.15.12:0/2820762171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac23fff640 1 -- 172.21.15.12:0/2820762171 >> v2:172.21.15.156:3300/0 conn(0x7fac2415df90 msgr2=0x7fac24161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac23fff640 1 --2- 172.21.15.12:0/2820762171 >> v2:172.21.15.156:3300/0 conn(0x7fac2415df90 0x7fac24161950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac23fff640 1 -- 172.21.15.12:0/2820762171 >> v1:172.21.15.156:6789/0 conn(0x7fac240cedf0 legacy=0x7fac240b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac23fff640 1 -- 172.21.15.12:0/2820762171 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac24162330 con 0x7fac240b9600 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac23fff640 1 --2- 172.21.15.12:0/2820762171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac240b9600 0x7fac240ce8b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fac2408d030 tx=0x7fac1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6458dba67a1760ce server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac237fe640 1 -- 172.21.15.12:0/2820762171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18034070 con 0x7fac240b9600 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac237fe640 1 -- 172.21.15.12:0/2820762171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18030860 con 0x7fac240b9600 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.397+0000 7fac237fe640 1 -- 172.21.15.12:0/2820762171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18030c30 con 0x7fac240b9600 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 -- 172.21.15.12:0/2820762171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac240b9600 msgr2=0x7fac240ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 --2- 172.21.15.12:0/2820762171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac240b9600 0x7fac240ce8b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fac2408d030 tx=0x7fac1802b9a0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 -- 172.21.15.12:0/2820762171 shutdown_connections 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 --2- 172.21.15.12:0/2820762171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac240b9600 0x7fac240ce8b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 --2- 172.21.15.12:0/2820762171 >> v2:172.21.15.156:3300/0 conn(0x7fac2415df90 0x7fac24161950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 -- 172.21.15.12:0/2820762171 >> 172.21.15.12:0/2820762171 conn(0x7fac2406c810 msgr2=0x7fac2406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 -- 172.21.15.12:0/2820762171 shutdown_connections 2024-03-08T08:31:35.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.398+0000 7fac2ac15640 1 -- 172.21.15.12:0/2820762171 wait complete. 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.399+0000 7fac2ac15640 1 Processor -- start 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.399+0000 7fac2ac15640 1 -- start start 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 0x7fac2414bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 0x7fac2415f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac240d3510 con 0x7fac2415df90 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac240b9a40 con 0x7fac240b9600 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac2ac15640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac240d3690 con 0x7fac240cedf0 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac291b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:35.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac291b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39374/0 (socket says 172.21.15.12:39374) 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac291b2640 1 -- 172.21.15.12:0/1690500097 learned_addr learned my addr 172.21.15.12:0/1690500097 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac289b1640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 0x7fac2414bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac23fff640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 0x7fac2415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac291b2640 1 -- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 msgr2=0x7fac2415f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.400+0000 7fac291b2640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 0x7fac2415f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac291b2640 1 -- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 msgr2=0x7fac2414bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac291b2640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 0x7fac2414bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac291b2640 1 -- 172.21.15.12:0/1690500097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2415fa80 con 0x7fac2415df90 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac289b1640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 0x7fac2414bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac23fff640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 0x7fac2415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac291b2640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c3c0 tx=0x7fac1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14017070 con 0x7fac2415df90 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac140035f0 con 0x7fac2415df90 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2415e4a0 con 0x7fac2415df90 2024-03-08T08:31:35.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac140039e0 con 0x7fac2415df90 2024-03-08T08:31:35.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.401+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2415ea60 con 0x7fac2415df90 2024-03-08T08:31:35.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.402+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac240749c0 con 0x7fac2415df90 2024-03-08T08:31:35.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.402+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac1401a070 con 0x7fac2415df90 2024-03-08T08:31:35.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.403+0000 7fac217fa640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:35.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.403+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fac140895e0 con 0x7fac2415df90 2024-03-08T08:31:35.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.403+0000 7fac289b1640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:35.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.403+0000 7fac289b1640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 0x7fac1006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac1c007eb0 tx=0x7fac1c0073c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:35.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.407+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fac14058420 con 0x7fac2415df90 2024-03-08T08:31:35.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:35.587+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac240d1710 con 0x7fac2415df90 2024-03-08T08:31:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.226+0000 7fac217fa640 1 -- 172.21.15.12:0/1690500097 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v39) v1 ==== 181+0+0 (secure 0 0 0) 0x7fac14057d20 con 0x7fac2415df90 2024-03-08T08:31:36.229 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 msgr2=0x7fac1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 0x7fac1006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac1c007eb0 tx=0x7fac1c0073c0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 msgr2=0x7fac24160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c3c0 tx=0x7fac1400c880 comp rx=0 tx=0).stop 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 shutdown_connections 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.229+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415df90 0x7fac24160380 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.230+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fac240cedf0 0x7fac2415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.230+0000 7fac2ac15640 1 --2- 172.21.15.12:0/1690500097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac240b9600 0x7fac2414bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.230+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 >> 172.21.15.12:0/1690500097 conn(0x7fac2406c810 msgr2=0x7fac240d3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:36.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.230+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 shutdown_connections 2024-03-08T08:31:36.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.230+0000 7fac2ac15640 1 -- 172.21.15.12:0/1690500097 wait complete. 2024-03-08T08:31:36.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-08T08:31:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.355+0000 7f7231b72640 1 Processor -- start 2024-03-08T08:31:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.355+0000 7f7231b72640 1 -- start start 2024-03-08T08:31:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f7231b72640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f7231b72640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f722c0b29a0 0x7f722c0d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f7231b72640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c0ce580 con 0x7f722c1623d0 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f7231b72640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c0ce860 con 0x7f722c0b29a0 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f7231b72640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0ceb40 con 0x7f722c15e050 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f722c0b29a0 0x7f722c0d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f722c0b29a0 0x7f722c0d1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39388/0 (socket says 172.21.15.12:39388) 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 -- 172.21.15.12:0/462788405 learned_addr learned my addr 172.21.15.12:0/462788405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 -- 172.21.15.12:0/462788405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 msgr2=0x7f722c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 --2- 172.21.15.12:0/462788405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 -- 172.21.15.12:0/462788405 >> v1:172.21.15.156:6789/0 conn(0x7f722c1623d0 legacy=0x7f722c0b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.356+0000 7f722bfff640 1 -- 172.21.15.12:0/462788405 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0cf340 con 0x7f722c0b29a0 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.357+0000 7f722bfff640 1 --2- 172.21.15.12:0/462788405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c0d1020 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7224009e50 tx=0x7f722402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e6d087a772d42 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.357+0000 7f722a7fc640 1 -- 172.21.15.12:0/462788405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722402c9e0 con 0x7f722c0b29a0 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.357+0000 7f722a7fc640 1 -- 172.21.15.12:0/462788405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722402cb80 con 0x7f722c0b29a0 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.357+0000 7f722a7fc640 1 -- 172.21.15.12:0/462788405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240343f0 con 0x7f722c0b29a0 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 -- 172.21.15.12:0/462788405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 msgr2=0x7f722c0d1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 --2- 172.21.15.12:0/462788405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c0d1020 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7224009e50 tx=0x7f722402be40 comp rx=0 tx=0).stop 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 -- 172.21.15.12:0/462788405 shutdown_connections 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 --2- 172.21.15.12:0/462788405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c0d1020 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 --2- 172.21.15.12:0/462788405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 -- 172.21.15.12:0/462788405 >> 172.21.15.12:0/462788405 conn(0x7f722c06c7c0 msgr2=0x7f722c06cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 -- 172.21.15.12:0/462788405 shutdown_connections 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.358+0000 7f7231b72640 1 -- 172.21.15.12:0/462788405 wait complete. 2024-03-08T08:31:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.359+0000 7f7231b72640 1 Processor -- start 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.359+0000 7f7231b72640 1 -- start start 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 0x7f722c1e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0b3200 con 0x7f722c0b29a0 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0b3080 con 0x7f722c1623d0 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0b5d10 con 0x7f722c15e050 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39398/0 (socket says 172.21.15.12:39398) 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 -- 172.21.15.12:0/145390289 learned_addr learned my addr 172.21.15.12:0/145390289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:36.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722bfff640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 0x7f722c1e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722affd640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 -- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 msgr2=0x7f722c0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0b4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 -- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 msgr2=0x7f722c1e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 0x7f722c1e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 -- 172.21.15.12:0/145390289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c15f4e0 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722bfff640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 0x7f722c1e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f722b7fe640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f722000b4a0 tx=0x7f722000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220014070 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7220010610 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220010a00 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.360+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c15f780 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.361+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c15fd40 con 0x7f722c0b29a0 2024-03-08T08:31:36.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.362+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c0749c0 con 0x7f722c0b29a0 2024-03-08T08:31:36.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.362+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7220017070 con 0x7f722c0b29a0 2024-03-08T08:31:36.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.362+0000 7f7213fff640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 0x7f721806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:36.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.362+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f7220089ba0 con 0x7f722c0b29a0 2024-03-08T08:31:36.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.363+0000 7f722affd640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 0x7f721806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:36.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.363+0000 7f722affd640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 0x7f721806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f722c0b9400 tx=0x7f721c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.367+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7220058a80 con 0x7f722c0b29a0 2024-03-08T08:31:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:36.553+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f722c15ca40 con 0x7f722c0b29a0 2024-03-08T08:31:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.230+0000 7f7213fff640 1 -- 172.21.15.12:0/145390289 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v40) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7220058380 con 0x7f722c0b29a0 2024-03-08T08:31:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-08T08:31:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 msgr2=0x7f721806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 0x7f721806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f722c0b9400 tx=0x7f721c008040 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 msgr2=0x7f722c1585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f722000b4a0 tx=0x7f722000b960 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 shutdown_connections 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f721806b4d0 0x7f721806d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f722c1623d0 0x7f722c1e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f722c15e050 0x7f722c0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 --2- 172.21.15.12:0/145390289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722c0b29a0 0x7f722c1585b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 >> 172.21.15.12:0/145390289 conn(0x7f722c06c7c0 msgr2=0x7f722c0b55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.234+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 shutdown_connections 2024-03-08T08:31:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.235+0000 7f7231b72640 1 -- 172.21.15.12:0/145390289 wait complete. 2024-03-08T08:31:37.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-08T08:31:37.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.361+0000 7f0352e2a640 1 Processor -- start 2024-03-08T08:31:37.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.361+0000 7f0352e2a640 1 -- start start 2024-03-08T08:31:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0352e2a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 0x7f034c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0352e2a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f034c0d25d0 0x7f034c0d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0352e2a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c0b6be0 con 0x7f034c15dec0 2024-03-08T08:31:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0352e2a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c0d2140 con 0x7f034c0d25d0 2024-03-08T08:31:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0352e2a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c15e2a0 con 0x7f034c0d0250 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f034bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 0x7f034c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f034c0d25d0 0x7f034c0d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f034c0d25d0 0x7f034c0d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:39556/0 (socket says 172.21.15.12:39556) 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f0350bc6640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f034c15dec0 0x7f034c161990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:35346/0 (socket says 172.21.15.12:35346) 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 -- 172.21.15.12:0/213560754 learned_addr learned my addr 172.21.15.12:0/213560754 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f034bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 0x7f034c0d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:41060/0 (socket says 172.21.15.12:41060) 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 -- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 msgr2=0x7f034c0d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 --2- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 0x7f034c0d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.362+0000 7f03513c7640 1 -- 172.21.15.12:0/213560754 >> v1:172.21.15.156:6789/0 conn(0x7f034c15dec0 legacy=0x7f034c161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.363+0000 7f03513c7640 1 -- 172.21.15.12:0/213560754 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c1620a0 con 0x7f034c0d25d0 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.363+0000 7f03513c7640 1 --2- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d25d0 0x7f034c0d2a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0344009e50 tx=0x7f034402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90fad298c2478b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.363+0000 7f034b7fe640 1 -- 172.21.15.12:0/213560754 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034402cb00 con 0x7f034c0d25d0 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f034b7fe640 1 -- 172.21.15.12:0/213560754 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034402cca0 con 0x7f034c0d25d0 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f034b7fe640 1 -- 172.21.15.12:0/213560754 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344034610 con 0x7f034c0d25d0 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 -- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d25d0 msgr2=0x7f034c0d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 --2- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d25d0 0x7f034c0d2a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0344009e50 tx=0x7f034402bf60 comp rx=0 tx=0).stop 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 -- 172.21.15.12:0/213560754 shutdown_connections 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 --2- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d25d0 0x7f034c0d2a60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 --2- 172.21.15.12:0/213560754 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c0d0250 0x7f034c0d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 -- 172.21.15.12:0/213560754 >> 172.21.15.12:0/213560754 conn(0x7f034c06c240 msgr2=0x7f034c06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 -- 172.21.15.12:0/213560754 shutdown_connections 2024-03-08T08:31:37.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.364+0000 7f0352e2a640 1 -- 172.21.15.12:0/213560754 wait complete. 2024-03-08T08:31:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.365+0000 7f0352e2a640 1 Processor -- start 2024-03-08T08:31:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.365+0000 7f0352e2a640 1 -- start start 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d0250 0x7f034c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c15dec0 0x7f034c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c162630 con 0x7f034c0d25d0 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c162350 con 0x7f034c0d0250 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0352e2a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0d1dd0 con 0x7f034c15dec0 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f034bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f034bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39406/0 (socket says 172.21.15.12:39406) 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f034bfff640 1 -- 172.21.15.12:0/3818936769 learned_addr learned my addr 172.21.15.12:0/3818936769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f03513c7640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c15dec0 0x7f034c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.366+0000 7f0350bc6640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d0250 0x7f034c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 -- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c15dec0 msgr2=0x7f034c15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c15dec0 0x7f034c15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 -- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d0250 msgr2=0x7f034c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d0250 0x7f034c0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 -- 172.21.15.12:0/3818936769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c0c9fd0 con 0x7f034c0d25d0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f034bfff640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f033800e9b0 tx=0x7f033800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338017070 con 0x7f034c0d25d0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033800c9a0 con 0x7f034c0d25d0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033800cd90 con 0x7f034c0d25d0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c145890 con 0x7f034c0d25d0 2024-03-08T08:31:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.367+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c145e50 con 0x7f034c0d25d0 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.368+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c0749c0 con 0x7f034c0d25d0 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.369+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f033801a070 con 0x7f034c0d25d0 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.369+0000 7f03497fa640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 0x7f033006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.369+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0338089ef0 con 0x7f034c0d25d0 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.369+0000 7f0350bc6640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 0x7f033006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.369+0000 7f0350bc6640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 0x7f033006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f034c0b62d0 tx=0x7f0340007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.373+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f034c0c9fd0 con 0x7f034c0d25d0 2024-03-08T08:31:37.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.557+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f034c0d1750 con 0x7f034c0d25d0 2024-03-08T08:31:37.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.560+0000 7f03497fa640 1 -- 172.21.15.12:0/3818936769 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0338059070 con 0x7f034c0d25d0 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.561+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 msgr2=0x7f033006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.561+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 0x7f033006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f034c0b62d0 tx=0x7f0340007ce0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 msgr2=0x7f034c15e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f033800e9b0 tx=0x7f033800ee70 comp rx=0 tx=0).stop 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 shutdown_connections 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f033006b500 0x7f033006d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f034c15dec0 0x7f034c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f034c0d25d0 0x7f034c15e420 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 --2- 172.21.15.12:0/3818936769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f034c0d0250 0x7f034c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 >> 172.21.15.12:0/3818936769 conn(0x7f034c06c240 msgr2=0x7f034c0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 shutdown_connections 2024-03-08T08:31:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.562+0000 7f0352e2a640 1 -- 172.21.15.12:0/3818936769 wait complete. 2024-03-08T08:31:37.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters' 2024-03-08T08:31:37.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.680+0000 7f0104684640 1 Processor -- start 2024-03-08T08:31:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.680+0000 7f0104684640 1 -- start start 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0104684640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f00fc099c00 0x7f00fc0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0104684640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00fc099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0104684640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc09a520 con 0x7f00fc0b24c0 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0104684640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc09ad80 con 0x7f00fc099c00 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0104684640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0cf570 con 0x7f00fc09a880 2024-03-08T08:31:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0102c21640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f00fc0b24c0 0x7f00fc0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34778/0 (socket says 172.21.15.12:34778) 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.681+0000 7f0102c21640 1 -- 172.21.15.12:0/1261134906 learned_addr learned my addr 172.21.15.12:0/1261134906 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 --2- 172.21.15.12:0/1261134906 >> v2:172.21.15.156:3300/0 conn(0x7f00fc099c00 0x7f00fc0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092466230 0 0) 0x7f00fc09a520 con 0x7f00fc0b24c0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0101c1f640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00fc099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e4003520 con 0x7f00fc0b24c0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638798377 0 0) 0x7f00e4003520 con 0x7f00fc0b24c0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 msgr2=0x7f00fc099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00fc099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 >> v2:172.21.15.156:3300/0 conn(0x7f00fc099c00 msgr2=0x7f00fc0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 --2- 172.21.15.12:0/1261134906 >> v2:172.21.15.156:3300/0 conn(0x7f00fc099c00 0x7f00fc0c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0cf850 con 0x7f00fc0b24c0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f00f4001670 con 0x7f00fc0b24c0 2024-03-08T08:31:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 >> v1:172.21.15.12:6789/0 conn(0x7f00fc0b24c0 legacy=0x7f00fc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00e4004d40 0x7f00e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc09ad80 con 0x7f00e40039b0 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0cf570 con 0x7f00e4004d40 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e400b9c0 con 0x7f00fc09a880 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0101c1f640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102c21640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00e4004d40 0x7f00e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 -- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 msgr2=0x7f00e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc09a880 0x7f00e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 -- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00e4004d40 msgr2=0x7f00e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00e4004d40 0x7f00e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.682+0000 7f0102420640 1 -- 172.21.15.12:0/1261134906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e4003520 con 0x7f00e40039b0 2024-03-08T08:31:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.683+0000 7f0102420640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00e4003dd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00e170 tx=0x7f00e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9e3f96e04b64e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.683+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec039070 con 0x7f00e40039b0 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.683+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec02cc20 con 0x7f00e40039b0 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.683+0000 7f010141e640 1 -- 172.21.15.12:0/1261134906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec009c10 con 0x7f00e40039b0 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 -- 172.21.15.12:0/1261134906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 msgr2=0x7f00e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 --2- 172.21.15.12:0/1261134906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00e4003dd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00e170 tx=0x7f00e4005830 comp rx=0 tx=0).stop 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0102c21640 1 -- 172.21.15.12:0/1261134906 reap_dead start 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 -- 172.21.15.12:0/1261134906 shutdown_connections 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 -- 172.21.15.12:0/1261134906 >> 172.21.15.12:0/1261134906 conn(0x7f00fc0b3490 msgr2=0x7f00fc0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 -- 172.21.15.12:0/1261134906 shutdown_connections 2024-03-08T08:31:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.684+0000 7f0104684640 1 -- 172.21.15.12:0/1261134906 wait complete. 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.685+0000 7f0104684640 1 Processor -- start 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.685+0000 7f0104684640 1 -- start start 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc0c4480 0x7f00fc070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00fc070f50 0x7f00fc15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0b2930 con 0x7f00e40039b0 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0ce0c0 con 0x7f00fc070f50 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0104684640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc06d0e0 con 0x7f00fc0c4480 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39424/0 (socket says 172.21.15.12:39424) 2024-03-08T08:31:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 -- 172.21.15.12:0/3105048783 learned_addr learned my addr 172.21.15.12:0/3105048783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102c21640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00fc070f50 0x7f00fc15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0101c1f640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc0c4480 0x7f00fc070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 -- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc0c4480 msgr2=0x7f00fc070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc0c4480 0x7f00fc070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 -- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00fc070f50 msgr2=0x7f00fc15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00fc070f50 0x7f00fc15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.686+0000 7f0102420640 1 -- 172.21.15.12:0/3105048783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc06c7b0 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f0102420640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00e290 tx=0x7f00ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec042020 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec03a040 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec039070 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc06ca90 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.687+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc0c48e0 con 0x7f00e40039b0 2024-03-08T08:31:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.688+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc0749c0 con 0x7f00e40039b0 2024-03-08T08:31:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.688+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00ec045020 con 0x7f00e40039b0 2024-03-08T08:31:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.689+0000 7f00faffd640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 0x7f00e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.689+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f00ec0abee0 con 0x7f00e40039b0 2024-03-08T08:31:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.689+0000 7f0101c1f640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 0x7f00e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.689+0000 7f0101c1f640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 0x7f00e806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00fc1f7380 tx=0x7f00f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.693+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f00ec04c030 con 0x7f00e40039b0 2024-03-08T08:31:37.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:37.941+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters"]} v 0) v1 -- 0x7f00fc0cc570 con 0x7f00e40039b0 2024-03-08T08:31:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.231+0000 7f00faffd640 1 -- 172.21.15.12:0/3105048783 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_admin.TestLabeledPerfCounters.test_per_client_labeled_perf_counters"]}]=0 v131) v1 ==== 164+0+0 (secure 0 0 0) 0x7f00ec07b1b0 con 0x7f00e40039b0 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.233+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 msgr2=0x7f00e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.233+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 0x7f00e806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00fc1f7380 tx=0x7f00f0008040 comp rx=0 tx=0).stop 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.233+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 msgr2=0x7f00fc0af0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00e290 tx=0x7f00ec009750 comp rx=0 tx=0).stop 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 shutdown_connections 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f00e806b4d0 0x7f00e806d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00fc070f50 0x7f00fc15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:38.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00fc0c4480 0x7f00fc070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:38.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 --2- 172.21.15.12:0/3105048783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00e40039b0 0x7f00fc0af0c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:38.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 >> 172.21.15.12:0/3105048783 conn(0x7f00fc0b3490 msgr2=0x7f00fc0b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:38.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 shutdown_connections 2024-03-08T08:31:38.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:38.234+0000 7f0104684640 1 -- 172.21.15.12:0/3105048783 wait complete. 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner:That the per-client labelled perf counters depict the clients ... ERROR 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner: 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner:ERROR: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner:That the per-client labelled perf counters depict the clients 2024-03-08T08:31:38.247 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa/tasks/cephfs/test_admin.py", line 70, in test_per_client_labeled_perf_counters 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: while proceed(): 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: raise MaxWhileTries(error_msg) 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:teuthology.exceptions.MaxWhileTries: 'wait for counters - client.4796' reached maximum tries (31) after waiting for 30 seconds 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:Ran 1 test in 122.022s 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner: 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:ERROR: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) 2024-03-08T08:31:38.248 INFO:tasks.cephfs_test_runner:That the per-client labelled perf counters depict the clients 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa/tasks/cephfs/test_admin.py", line 70, in test_per_client_labeled_perf_counters 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner: while proceed(): 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner: raise MaxWhileTries(error_msg) 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner:teuthology.exceptions.MaxWhileTries: 'wait for counters - client.4796' reached maximum tries (31) after waiting for 30 seconds 2024-03-08T08:31:38.249 INFO:tasks.cephfs_test_runner: 2024-03-08T08:31:38.249 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) 2024-03-08T08:31:38.444 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f88257a81fbd4caf8a2a4dc70d094848 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_joscollin_ceph_f6f80007dcf9e039bb253dd50a92ba3c71c6943a/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters) 2024-03-08T08:31:38.446 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-08T08:31:38.455 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-08T08:31:38.590 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-08T08:31:38.591 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:31:38.591 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:38.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=208 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-08T08:31:38.611 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-08T08:31:38.611 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:38.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=208 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-08T08:31:38.667 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:31:38.667 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:38.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-08T08:31:38.686 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-08T08:31:38.686 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T08:31:38.743 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-08T08:31:38.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:38.744 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-08T08:31:38.760 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-08T08:31:38.760 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-08T08:31:38.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:38.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-08T08:31:38.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:38.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T08:31:38.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:38.900 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-08T08:31:38.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:38.994 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-08T08:31:39.012 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-08T08:31:39.013 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-08T08:31:39.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:39.013 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-08T08:31:39.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:39.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-08T08:31:39.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T08:31:39.154 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-03-08T08:31:39.256 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-08T08:31:39.265 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-08T08:31:39.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:31:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.372+0000 7fc10d69a640 1 Processor -- start 2024-03-08T08:31:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.372+0000 7fc10d69a640 1 -- start start 2024-03-08T08:31:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc10d69a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc108143ae0 0x7fc10806e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc10d69a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10806f070 con 0x7fc10806eb30 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc10d69a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10815e510 con 0x7fc108143ec0 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc10d69a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10815e7f0 con 0x7fc108143ae0 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc1067fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc10806eb30 0x7fc1080cebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36978/0 (socket says 172.21.15.12:36978) 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc1067fc640 1 -- 172.21.15.12:0/1731292176 learned_addr learned my addr 172.21.15.12:0/1731292176 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc106ffd640 1 --2- 172.21.15.12:0/1731292176 >> v2:172.21.15.156:3300/0 conn(0x7fc108143ae0 0x7fc10806e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401712362 0 0) 0x7fc10806f070 con 0x7fc10806eb30 2024-03-08T08:31:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f0003520 con 0x7fc10806eb30 2024-03-08T08:31:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384964409 0 0) 0x7fc0f0003520 con 0x7fc10806eb30 2024-03-08T08:31:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc106ffd640 1 -- 172.21.15.12:0/1731292176 >> v1:172.21.15.156:6789/0 conn(0x7fc108143ec0 legacy=0x7fc1080b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc106ffd640 1 -- 172.21.15.12:0/1731292176 >> v1:172.21.15.12:6789/0 conn(0x7fc10806eb30 legacy=0x7fc1080cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.373+0000 7fc106ffd640 1 -- 172.21.15.12:0/1731292176 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10815ead0 con 0x7fc108143ae0 2024-03-08T08:31:39.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.374+0000 7fc106ffd640 1 --2- 172.21.15.12:0/1731292176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10806e5f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80097a0 tx=0x7fc0f802b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d53e62bb1091a647 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.374+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8037070 con 0x7fc108143ae0 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.374+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8030510 con 0x7fc108143ae0 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.374+0000 7fc105ffb640 1 -- 172.21.15.12:0/1731292176 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc001670 con 0x7fc108143ae0 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 -- 172.21.15.12:0/1731292176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 msgr2=0x7fc10806e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 --2- 172.21.15.12:0/1731292176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10806e5f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80097a0 tx=0x7fc0f802b680 comp rx=0 tx=0).stop 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 -- 172.21.15.12:0/1731292176 shutdown_connections 2024-03-08T08:31:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 --2- 172.21.15.12:0/1731292176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10806e5f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 -- 172.21.15.12:0/1731292176 >> 172.21.15.12:0/1731292176 conn(0x7fc10806abd0 msgr2=0x7fc10806afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 -- 172.21.15.12:0/1731292176 shutdown_connections 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.375+0000 7fc10d69a640 1 -- 172.21.15.12:0/1731292176 wait complete. 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.376+0000 7fc10d69a640 1 Processor -- start 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.376+0000 7fc10d69a640 1 -- start start 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc10806eb30 0x7fc1081f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10813b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc108144560 con 0x7fc108143ec0 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815f2d0 con 0x7fc108143ae0 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc10d69a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081446e0 con 0x7fc10806eb30 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54502/0 (socket says 172.21.15.12:54502) 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 -- 172.21.15.12:0/4041018048 learned_addr learned my addr 172.21.15.12:0/4041018048 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1067fc640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10813b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc106ffd640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc10806eb30 0x7fc1081f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 -- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc10806eb30 msgr2=0x7fc1081f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc10806eb30 0x7fc1081f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 -- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 msgr2=0x7fc10813b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10813b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 -- 172.21.15.12:0/4041018048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc108139da0 con 0x7fc108143ec0 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.377+0000 7fc1077fe640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c840 tx=0x7fc10000cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.378+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100017070 con 0x7fc108143ec0 2024-03-08T08:31:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.378+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc100003da0 con 0x7fc108143ec0 2024-03-08T08:31:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.378+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10813ba80 con 0x7fc108143ec0 2024-03-08T08:31:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.378+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100005290 con 0x7fc108143ec0 2024-03-08T08:31:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.378+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10813c040 con 0x7fc108143ec0 2024-03-08T08:31:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.379+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1080b2fc0 con 0x7fc108143ec0 2024-03-08T08:31:39.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.379+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc10001a070 con 0x7fc108143ec0 2024-03-08T08:31:39.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.379+0000 7fc0ef7fe640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 0x7fc0f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.380+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc100089580 con 0x7fc108143ec0 2024-03-08T08:31:39.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.380+0000 7fc106ffd640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 0x7fc0f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.380+0000 7fc106ffd640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 0x7fc0f406d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8034000 tx=0x7fc0f8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.384+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc100058140 con 0x7fc108143ec0 2024-03-08T08:31:39.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.559+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1080b2220 con 0x7fc0f406b4d0 2024-03-08T08:31:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.559+0000 7fc0ef7fe640 1 -- 172.21.15.12:0/4041018048 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13479 (secure 0 0 0) 0x7fc1080b2220 con 0x7fc0f406b4d0 2024-03-08T08:31:39.561 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:39.562 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:31:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.561+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 msgr2=0x7fc0f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.561+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 0x7fc0f406d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8034000 tx=0x7fc0f8008030 comp rx=0 tx=0).stop 2024-03-08T08:31:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 msgr2=0x7fc1081387d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c840 tx=0x7fc10000cd00 comp rx=0 tx=0).stop 2024-03-08T08:31:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 shutdown_connections 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fc0f406b4d0 0x7fc0f406d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc108143ec0 0x7fc1081387d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc108143ae0 0x7fc10813b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 --2- 172.21.15.12:0/4041018048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc10806eb30 0x7fc1081f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 >> 172.21.15.12:0/4041018048 conn(0x7fc10806abd0 msgr2=0x7fc1080b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 shutdown_connections 2024-03-08T08:31:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.562+0000 7fc10d69a640 1 -- 172.21.15.12:0/4041018048 wait complete. 2024-03-08T08:31:39.574 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-03-08T08:31:39.025973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261712,"kb_used_data":8100,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713008,"statfs":{"total":773094113280,"available":772826120192,"internally_reserved":0,"allocated":8294400,"data_stored":4307042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5876414,"num_objects":-1023,"num_object_clones":0,"num_object_copies":-2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2091,"num_write_kb":-6982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-64,"num_bytes_recovered":-187,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001284"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-08T08:31:34.504871+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:31:34.504871+0000","last_peered":"2024-03-08T08:31:34.504871+0000","last_clean":"2024-03-08T08:31:34.504871+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:31:34.504871+0000","last_undegraded":"2024-03-08T08:31:34.504871+0000","last_fullsized":"2024-03-08T08:31:34.504871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1978368,"data_stored":763885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35800,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38792,"kb_used_data":3456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333048,"statfs":{"total":96636764160,"available":96597041152,"internally_reserved":0,"allocated":3538944,"data_stored":2436007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39952,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:31:39.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:31:39.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.674+0000 7f141d023640 1 Processor -- start 2024-03-08T08:31:39.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.674+0000 7f141d023640 1 -- start start 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141d023640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14180c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141d023640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f14180d0d30 0x7f14180cb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141d023640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180cf870 con 0x7f14180cf410 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141d023640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141815ae50 con 0x7f14180d0d30 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141d023640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cfb10 con 0x7f141815aa70 2024-03-08T08:31:39.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f14180cf410 0x7f14180d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36992/0 (socket says 172.21.15.12:36992) 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141659c640 1 -- 172.21.15.12:0/1874500033 learned_addr learned my addr 172.21.15.12:0/1874500033 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f141759e640 1 --2- 172.21.15.12:0/1874500033 >> v2:172.21.15.156:3300/0 conn(0x7f14180d0d30 0x7f14180cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f1416d9d640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14180c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015303856 0 0) 0x7f14180cf870 con 0x7f14180cf410 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.675+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1404003520 con 0x7f14180cf410 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f141759e640 1 -- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 msgr2=0x7f14180c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f141759e640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14180c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105259210 0 0) 0x7f1404003520 con 0x7f14180cf410 2024-03-08T08:31:39.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f141759e640 1 -- 172.21.15.12:0/1874500033 >> v1:172.21.15.12:6789/0 conn(0x7f14180cf410 legacy=0x7f14180d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f141759e640 1 -- 172.21.15.12:0/1874500033 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180cfdf0 con 0x7f14180d0d30 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.676+0000 7f141759e640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f14180cb580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1408009d90 tx=0x7f140802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dfb83641ef2b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408039070 con 0x7f14180d0d30 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140802c8a0 con 0x7f14180d0d30 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f1415d9b640 1 -- 172.21.15.12:0/1874500033 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400001670 con 0x7f14180d0d30 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 -- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 msgr2=0x7f14180cb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f14180cb580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1408009d90 tx=0x7f140802bd70 comp rx=0 tx=0).stop 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 -- 172.21.15.12:0/1874500033 shutdown_connections 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f14180cb580 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 --2- 172.21.15.12:0/1874500033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14180c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 -- 172.21.15.12:0/1874500033 >> 172.21.15.12:0/1874500033 conn(0x7f1418099a40 msgr2=0x7f1418099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 -- 172.21.15.12:0/1874500033 shutdown_connections 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.677+0000 7f141d023640 1 -- 172.21.15.12:0/1874500033 wait complete. 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.678+0000 7f141d023640 1 Processor -- start 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 -- start start 2024-03-08T08:31:39.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f1418154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14181f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cbac0 con 0x7f14180cf410 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cb060 con 0x7f14180d0d30 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141d023640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cbc40 con 0x7f141815aa70 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f1416d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f1416d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54514/0 (socket says 172.21.15.12:54514) 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f1416d9d640 1 -- 172.21.15.12:0/3546974515 learned_addr learned my addr 172.21.15.12:0/3546974515 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141659c640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f1418154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.679+0000 7f141759e640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14181f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 -- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 msgr2=0x7f14181f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14181f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 -- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 msgr2=0x7f1418154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f1418154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 -- 172.21.15.12:0/3546974515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1418155290 con 0x7f14180cf410 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f141659c640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f1418154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f1416d9d640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f141000e8a0 tx=0x7f141000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410017070 con 0x7f14180cf410 2024-03-08T08:31:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000c940 con 0x7f14180cf410 2024-03-08T08:31:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14181f7120 con 0x7f14180cf410 2024-03-08T08:31:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141000cd30 con 0x7f14180cf410 2024-03-08T08:31:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.680+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14180b62f0 con 0x7f14180cf410 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.681+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14180749c0 con 0x7f14180cf410 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.682+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f141001a070 con 0x7f14180cf410 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.682+0000 7f140f7fe640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.682+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1410089ff0 con 0x7f14180cf410 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.682+0000 7f141659c640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.682+0000 7f141659c640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 0x7f13f806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14181595a0 tx=0x7f1400006ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.686+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1410058bb0 con 0x7f14180cf410 2024-03-08T08:31:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.866+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14180d1680 con 0x7f13f806b500 2024-03-08T08:31:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.866+0000 7f140f7fe640 1 -- 172.21.15.12:0/3546974515 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13479 (secure 0 0 0) 0x7f14180d1680 con 0x7f13f806b500 2024-03-08T08:31:39.868 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:31:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 msgr2=0x7f13f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 0x7f13f806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14181595a0 tx=0x7f1400006ac0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 msgr2=0x7f141806b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f141000e8a0 tx=0x7f141000ed60 comp rx=0 tx=0).stop 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 shutdown_connections 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f141815aa70 0x7f14181f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14180d0d30 0x7f1418154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 --2- 172.21.15.12:0/3546974515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14180cf410 0x7f141806b710 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.868+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 >> 172.21.15.12:0/3546974515 conn(0x7f1418099a40 msgr2=0x7f14180d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.869+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 shutdown_connections 2024-03-08T08:31:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.869+0000 7f141d023640 1 -- 172.21.15.12:0/3546974515 wait complete. 2024-03-08T08:31:39.881 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-03-08T08:31:39.025973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261712,"kb_used_data":8100,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713008,"statfs":{"total":773094113280,"available":772826120192,"internally_reserved":0,"allocated":8294400,"data_stored":4307042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5876414,"num_objects":-1023,"num_object_clones":0,"num_object_copies":-2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2091,"num_write_kb":-6982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-64,"num_bytes_recovered":-187,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001284"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-08T08:31:34.504871+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:31:34.504871+0000","last_peered":"2024-03-08T08:31:34.504871+0000","last_clean":"2024-03-08T08:31:34.504871+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:31:34.504871+0000","last_undegraded":"2024-03-08T08:31:34.504871+0000","last_fullsized":"2024-03-08T08:31:34.504871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1978368,"data_stored":763885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35800,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38792,"kb_used_data":3456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333048,"statfs":{"total":96636764160,"available":96597041152,"internally_reserved":0,"allocated":3538944,"data_stored":2436007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39952,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:31:39.881 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-08T08:31:39.882 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:31:39.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.981+0000 7f090fd93640 1 Processor -- start 2024-03-08T08:31:39.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.981+0000 7f090fd93640 1 -- start start 2024-03-08T08:31:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090fd93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 0x7f090815c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090fd93640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f090806b2e0 0x7f09080d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090fd93640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080cd3b0 con 0x7f09080b0fc0 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090fd93640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09080cd690 con 0x7f090806b2e0 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090fd93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09080cd970 con 0x7f09080b0be0 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090d32e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f09080b0fc0 0x7f090806abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37006/0 (socket says 172.21.15.12:37006) 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.982+0000 7f090d32e640 1 -- 172.21.15.12:0/1251317366 learned_addr learned my addr 172.21.15.12:0/1251317366 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090e330640 1 --2- 172.21.15.12:0/1251317366 >> v2:172.21.15.156:3300/0 conn(0x7f090806b2e0 0x7f09080d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090db2f640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 0x7f090815c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090cb2d640 1 -- 172.21.15.12:0/1251317366 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049063375 0 0) 0x7f09080cd3b0 con 0x7f09080b0fc0 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090cb2d640 1 -- 172.21.15.12:0/1251317366 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0003520 con 0x7f09080b0fc0 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090e330640 1 -- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 msgr2=0x7f090815c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090e330640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 0x7f090815c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090e330640 1 -- 172.21.15.12:0/1251317366 >> v1:172.21.15.12:6789/0 conn(0x7f09080b0fc0 legacy=0x7f090806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090e330640 1 -- 172.21.15.12:0/1251317366 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080cdc50 con 0x7f090806b2e0 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.983+0000 7f090db2f640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 0x7f090815c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:39.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090e330640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f090806b2e0 0x7f09080d0b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f090002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea1cc5726573f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090cb2d640 1 -- 172.21.15.12:0/1251317366 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900035020 con 0x7f090806b2e0 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090cb2d640 1 -- 172.21.15.12:0/1251317366 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0900031830 con 0x7f090806b2e0 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090cb2d640 1 -- 172.21.15.12:0/1251317366 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900031c00 con 0x7f090806b2e0 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 -- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f090806b2e0 msgr2=0x7f09080d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f090806b2e0 0x7f09080d0b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f090002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 -- 172.21.15.12:0/1251317366 shutdown_connections 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f090806b2e0 0x7f09080d0b30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 --2- 172.21.15.12:0/1251317366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09080b0be0 0x7f090815c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 -- 172.21.15.12:0/1251317366 >> 172.21.15.12:0/1251317366 conn(0x7f0908099a40 msgr2=0x7f0908099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.984+0000 7f090fd93640 1 -- 172.21.15.12:0/1251317366 shutdown_connections 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.985+0000 7f090fd93640 1 -- 172.21.15.12:0/1251317366 wait complete. 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 Processor -- start 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 -- start start 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f090806b2e0 0x7f0908144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09080b0be0 0x7f09080aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090815cee0 con 0x7f09080b0fc0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080b18c0 con 0x7f09080b0be0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090fd93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09080ccf60 con 0x7f090806b2e0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090e330640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090e330640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54528/0 (socket says 172.21.15.12:54528) 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090e330640 1 -- 172.21.15.12:0/510788333 learned_addr learned my addr 172.21.15.12:0/510788333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090d32e640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09080b0be0 0x7f09080aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.986+0000 7f090db2f640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f090806b2e0 0x7f0908144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 -- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f090806b2e0 msgr2=0x7f0908144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f090806b2e0 0x7f0908144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 -- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09080b0be0 msgr2=0x7f09080aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09080b0be0 0x7f09080aee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 -- 172.21.15.12:0/510788333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080af3c0 con 0x7f09080b0fc0 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090e330640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f09000098c0 tx=0x7f09000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900035040 con 0x7f09080b0fc0 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090003b840 con 0x7f09080b0fc0 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081f4cf0 con 0x7f09080b0fc0 2024-03-08T08:31:39.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090003bc30 con 0x7f09080b0fc0 2024-03-08T08:31:39.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.987+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09081f52b0 con 0x7f09080b0fc0 2024-03-08T08:31:39.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.988+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09080749c0 con 0x7f09080b0fc0 2024-03-08T08:31:39.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.989+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0900038020 con 0x7f09080b0fc0 2024-03-08T08:31:39.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.989+0000 7f09067fc640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 0x7f08f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:39.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.989+0000 7f090db2f640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 0x7f08f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:39.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.989+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09000ab960 con 0x7f09080b0fc0 2024-03-08T08:31:39.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.989+0000 7f090db2f640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 0x7f08f406d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09080b0940 tx=0x7f08f800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:39.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:39.994+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f090007a490 con 0x7f09080b0fc0 2024-03-08T08:31:40.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.170+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09080b6760 con 0x7f08f406b500 2024-03-08T08:31:40.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.170+0000 7f09067fc640 1 -- 172.21.15.12:0/510788333 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13479 (secure 0 0 0) 0x7f09080b6760 con 0x7f08f406b500 2024-03-08T08:31:40.172 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:40.172 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 msgr2=0x7f08f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 0x7f08f406d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09080b0940 tx=0x7f08f800b040 comp rx=0 tx=0).stop 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 msgr2=0x7f0908150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f09000098c0 tx=0x7f09000046c0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 shutdown_connections 2024-03-08T08:31:40.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f08f406b500 0x7f08f406d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09080b0fc0 0x7f0908150ef0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09080b0be0 0x7f09080aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 --2- 172.21.15.12:0/510788333 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f090806b2e0 0x7f0908144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 >> 172.21.15.12:0/510788333 conn(0x7f0908099a40 msgr2=0x7f090806abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 shutdown_connections 2024-03-08T08:31:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.172+0000 7f090fd93640 1 -- 172.21.15.12:0/510788333 wait complete. 2024-03-08T08:31:40.184 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-03-08T08:31:39.025973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261712,"kb_used_data":8100,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713008,"statfs":{"total":773094113280,"available":772826120192,"internally_reserved":0,"allocated":8294400,"data_stored":4307042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5876414,"num_objects":-1023,"num_object_clones":0,"num_object_copies":-2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2091,"num_write_kb":-6982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-64,"num_bytes_recovered":-187,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001284"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-08T08:31:34.504871+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:31:34.504871+0000","last_peered":"2024-03-08T08:31:34.504871+0000","last_clean":"2024-03-08T08:31:34.504871+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:31:34.504871+0000","last_undegraded":"2024-03-08T08:31:34.504871+0000","last_fullsized":"2024-03-08T08:31:34.504871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1978368,"data_stored":763885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35800,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38792,"kb_used_data":3456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333048,"statfs":{"total":96636764160,"available":96597041152,"internally_reserved":0,"allocated":3538944,"data_stored":2436007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39952,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:31:40.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T08:31:40.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.285+0000 7fdbfa92c640 1 Processor -- start 2024-03-08T08:31:40.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.285+0000 7fdbfa92c640 1 -- start start 2024-03-08T08:31:40.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.286+0000 7fdbfa92c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdbf415f1a0 0x7fdbf40cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.286+0000 7fdbfa92c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf406cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.286+0000 7fdbfa92c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf40d0110 con 0x7fdbf4099c00 2024-03-08T08:31:40.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.286+0000 7fdbfa92c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf415f580 con 0x7fdbf415f1a0 2024-03-08T08:31:40.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.286+0000 7fdbfa92c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf40d03f0 con 0x7fdbf40d11b0 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdbf415f1a0 0x7fdbf40cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf406cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf8ec9640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fdbf4099c00 0x7fdbf40cbff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:36490/0 (socket says 172.21.15.12:36490) 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf8ec9640 1 -- 172.21.15.12:0/3587739326 learned_addr learned my addr 172.21.15.12:0/3587739326 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf3fff640 1 -- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 msgr2=0x7fdbf406cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf3fff640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf406cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf3fff640 1 -- 172.21.15.12:0/3587739326 >> v1:172.21.15.156:6789/0 conn(0x7fdbf4099c00 legacy=0x7fdbf40cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf3fff640 1 -- 172.21.15.12:0/3587739326 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf40d06d0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf2ffd640 1 -- 172.21.15.12:0/3587739326 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497909387 0 0) 0x7fdbf40d0110 con 0x7fdbf4099c00 2024-03-08T08:31:40.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.287+0000 7fdbf37fe640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf406cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbf3fff640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cfb10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40097a0 tx=0x7fdbe402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46244ff3db6a056e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbf2ffd640 1 -- 172.21.15.12:0/3587739326 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4035020 con 0x7fdbf415f1a0 2024-03-08T08:31:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbf2ffd640 1 -- 172.21.15.12:0/3587739326 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe4031810 con 0x7fdbf415f1a0 2024-03-08T08:31:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbf2ffd640 1 -- 172.21.15.12:0/3587739326 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4031be0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 -- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf415f1a0 msgr2=0x7fdbf40cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cfb10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40097a0 tx=0x7fdbe402b8b0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 -- 172.21.15.12:0/3587739326 shutdown_connections 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf406cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/3587739326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cfb10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 -- 172.21.15.12:0/3587739326 >> 172.21.15.12:0/3587739326 conn(0x7fdbf40b3490 msgr2=0x7fdbf40b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 -- 172.21.15.12:0/3587739326 shutdown_connections 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.288+0000 7fdbfa92c640 1 -- 172.21.15.12:0/3587739326 wait complete. 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.289+0000 7fdbfa92c640 1 Processor -- start 2024-03-08T08:31:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 -- start start 2024-03-08T08:31:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf40c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf406bf00 con 0x7fdbf415f1a0 2024-03-08T08:31:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf40d0c90 con 0x7fdbf4099c00 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.290+0000 7fdbfa92c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf409a070 con 0x7fdbf40d11b0 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:49662/0 (socket says 172.21.15.12:49662) 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf3fff640 1 -- 172.21.15.12:0/230221819 learned_addr learned my addr 172.21.15.12:0/230221819 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf37fe640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf40c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 -- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 msgr2=0x7fdbf40c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf40c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 -- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 msgr2=0x7fdbf414df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 -- 172.21.15.12:0/230221819 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf40c92c0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf3fff640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf37fe640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf40c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf8ec9640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cab20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c6d0 tx=0x7fdbec00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec018070 con 0x7fdbf415f1a0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec004d60 con 0x7fdbf415f1a0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf40c95a0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00a880 con 0x7fdbf415f1a0 2024-03-08T08:31:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.291+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf40c9b60 con 0x7fdbf415f1a0 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.292+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf40749c0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.293+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbec01b070 con 0x7fdbf415f1a0 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.293+0000 7fdbf0ff9640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 0x7fdbe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.293+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdbec0895c0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.293+0000 7fdbf3fff640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 0x7fdbe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:40.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.294+0000 7fdbf3fff640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 0x7fdbe006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdbe402b8f0 tx=0x7fdbe4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.298+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdbec0580f0 con 0x7fdbf415f1a0 2024-03-08T08:31:40.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.479+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf40b7920 con 0x7fdbf415f1a0 2024-03-08T08:31:40.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.480+0000 7fdbf0ff9640 1 -- 172.21.15.12:0/230221819 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+13020 (secure 0 0 0) 0x7fdbec05c190 con 0x7fdbf415f1a0 2024-03-08T08:31:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:40.481 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":40,"fsid":"e255a89c-5cb8-4b01-8b38-e7c34c7e2a9c","created":"2024-03-08T08:28:48.656415+0000","modified":"2024-03-08T08:31:37.227184+0000","last_up_change":"2024-03-08T08:28:59.148406+0000","last_in_change":"2024-03-08T08:28:50.825188+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T08:28:59.035221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b18ed71d-fb54-456e-b684-4eed2d74d441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6812","nonce":1155150761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6815","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6818","nonce":1155150761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6831","nonce":1155150761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1155150761},{"type":"v1","addr":"172.21.15.12:6825","nonce":1155150761}]},"public_addr":"172.21.15.12:6812/1155150761","cluster_addr":"172.21.15.12:6818/1155150761","heartbeat_back_addr":"172.21.15.12:6831/1155150761","heartbeat_front_addr":"172.21.15.12:6825/1155150761","state":["exists","up"]},{"osd":1,"uuid":"1a7daaf5-7d2b-4593-8fb6-7f73346f63a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6811","nonce":1692589129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6817","nonce":1692589129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6827","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6830","nonce":1692589129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":1692589129},{"type":"v1","addr":"172.21.15.12:6824","nonce":1692589129}]},"public_addr":"172.21.15.12:6811/1692589129","cluster_addr":"172.21.15.12:6817/1692589129","heartbeat_back_addr":"172.21.15.12:6830/1692589129","heartbeat_front_addr":"172.21.15.12:6824/1692589129","state":["exists","up"]},{"osd":2,"uuid":"f11ad7d7-d93b-4dbf-9c65-91a9433a5095","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6801","nonce":2701699187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6803","nonce":2701699187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6807","nonce":2701699187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2701699187},{"type":"v1","addr":"172.21.15.12:6805","nonce":2701699187}]},"public_addr":"172.21.15.12:6801/2701699187","cluster_addr":"172.21.15.12:6803/2701699187","heartbeat_back_addr":"172.21.15.12:6807/2701699187","heartbeat_front_addr":"172.21.15.12:6805/2701699187","state":["exists","up"]},{"osd":3,"uuid":"f3e67c25-59c6-4c91-aa05-7e2310962ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6813","nonce":2394545747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6819","nonce":2394545747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6829","nonce":2394545747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2394545747},{"type":"v1","addr":"172.21.15.12:6823","nonce":2394545747}]},"public_addr":"172.21.15.12:6813/2394545747","cluster_addr":"172.21.15.12:6819/2394545747","heartbeat_back_addr":"172.21.15.12:6829/2394545747","heartbeat_front_addr":"172.21.15.12:6823/2394545747","state":["exists","up"]},{"osd":4,"uuid":"cd571d9f-d56f-4013-95ef-1d1b69a3a852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6817","nonce":602210282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6819","nonce":602210282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6823","nonce":602210282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":602210282},{"type":"v1","addr":"172.21.15.156:6821","nonce":602210282}]},"public_addr":"172.21.15.156:6817/602210282","cluster_addr":"172.21.15.156:6819/602210282","heartbeat_back_addr":"172.21.15.156:6823/602210282","heartbeat_front_addr":"172.21.15.156:6821/602210282","state":["exists","up"]},{"osd":5,"uuid":"45e09db2-d6a1-46dd-b7c1-4436bd779079","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6825","nonce":1688693160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6827","nonce":1688693160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6831","nonce":1688693160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1688693160},{"type":"v1","addr":"172.21.15.156:6829","nonce":1688693160}]},"public_addr":"172.21.15.156:6825/1688693160","cluster_addr":"172.21.15.156:6827/1688693160","heartbeat_back_addr":"172.21.15.156:6831/1688693160","heartbeat_front_addr":"172.21.15.156:6829/1688693160","state":["exists","up"]},{"osd":6,"uuid":"01b92c46-4426-4ab9-b1d2-d385d65ebdc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6801","nonce":2822268422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6803","nonce":2822268422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6807","nonce":2822268422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2822268422},{"type":"v1","addr":"172.21.15.156:6805","nonce":2822268422}]},"public_addr":"172.21.15.156:6801/2822268422","cluster_addr":"172.21.15.156:6803/2822268422","heartbeat_back_addr":"172.21.15.156:6807/2822268422","heartbeat_front_addr":"172.21.15.156:6805/2822268422","state":["exists","up"]},{"osd":7,"uuid":"3ddbf98e-92c1-4352-8ebf-f1447907b04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6809","nonce":1139145947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6811","nonce":1139145947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6815","nonce":1139145947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1139145947},{"type":"v1","addr":"172.21.15.156:6813","nonce":1139145947}]},"public_addr":"172.21.15.156:6809/1139145947","cluster_addr":"172.21.15.156:6811/1139145947","heartbeat_back_addr":"172.21.15.156:6815/1139145947","heartbeat_front_addr":"172.21.15.156:6813/1139145947","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.946288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.986458+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:56.011134+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.913617+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T08:28:55.971179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6837/2820296658":"2024-03-09T08:31:34.219751+0000","172.21.15.12:6836/2820296658":"2024-03-09T08:31:34.219751+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T08:31:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 msgr2=0x7fdbe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 0x7fdbe006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdbe402b8f0 tx=0x7fdbe4008030 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 msgr2=0x7fdbf40cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cab20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c6d0 tx=0x7fdbec00cb90 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 shutdown_connections 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fdbe006b4d0 0x7fdbe006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbf415f1a0 0x7fdbf40cab20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdbf40d11b0 0x7fdbf40c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 --2- 172.21.15.12:0/230221819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbf4099c00 0x7fdbf414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 >> 172.21.15.12:0/230221819 conn(0x7fdbf40b3490 msgr2=0x7fdbf40ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:40.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 shutdown_connections 2024-03-08T08:31:40.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:40.482+0000 7fdbfa92c640 1 -- 172.21.15.12:0/230221819 wait complete. 2024-03-08T08:31:41.495 INFO:tasks.ceph:Scrubbing osd.0 2024-03-08T08:31:41.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.611+0000 7f59e9519640 1 Processor -- start 2024-03-08T08:31:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.611+0000 7f59e9519640 1 -- start start 2024-03-08T08:31:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e9519640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e9519640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59e40ceba0 0x7f59e40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e9519640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e415b490 con 0x7f59e40cd390 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e9519640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e40cd770 con 0x7f59e40ceba0 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e9519640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e415b770 con 0x7f59e415aeb0 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59e40ceba0 0x7f59e40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59e40ceba0 0x7f59e40cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54540/0 (socket says 172.21.15.12:54540) 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 -- 172.21.15.12:0/4137941248 learned_addr learned my addr 172.21.15.12:0/4137941248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e27fc640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 -- 172.21.15.12:0/4137941248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 msgr2=0x7f59e40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 -- 172.21.15.12:0/4137941248 >> v1:172.21.15.156:6789/0 conn(0x7f59e40cd390 legacy=0x7f59e406eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.612+0000 7f59e37fe640 1 -- 172.21.15.12:0/4137941248 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e415ba50 con 0x7f59e40ceba0 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.613+0000 7f59e37fe640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40cf010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f59d400d150 tx=0x7f59d402e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82732bb924ac9fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.613+0000 7f59e1ffb640 1 -- 172.21.15.12:0/4137941248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d400b0d0 con 0x7f59e40ceba0 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.613+0000 7f59e1ffb640 1 -- 172.21.15.12:0/4137941248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d400b270 con 0x7f59e40ceba0 2024-03-08T08:31:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.613+0000 7f59e1ffb640 1 -- 172.21.15.12:0/4137941248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4009600 con 0x7f59e40ceba0 2024-03-08T08:31:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 -- 172.21.15.12:0/4137941248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 msgr2=0x7f59e40cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40cf010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f59d400d150 tx=0x7f59d402e160 comp rx=0 tx=0).stop 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 -- 172.21.15.12:0/4137941248 shutdown_connections 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40cf010 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 --2- 172.21.15.12:0/4137941248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 -- 172.21.15.12:0/4137941248 >> 172.21.15.12:0/4137941248 conn(0x7f59e406abd0 msgr2=0x7f59e406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 -- 172.21.15.12:0/4137941248 shutdown_connections 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.614+0000 7f59e9519640 1 -- 172.21.15.12:0/4137941248 wait complete. 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.615+0000 7f59e9519640 1 Processor -- start 2024-03-08T08:31:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.615+0000 7f59e9519640 1 -- start start 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59e40cd390 0x7f59e414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40b3760 con 0x7f59e40ceba0 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40cf550 con 0x7f59e40cd390 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e9519640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40b38e0 con 0x7f59e415aeb0 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54546/0 (socket says 172.21.15.12:54546) 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e27fc640 1 -- 172.21.15.12:0/1829428494 learned_addr learned my addr 172.21.15.12:0/1829428494 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e37fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e2ffd640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59e40cd390 0x7f59e414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.616+0000 7f59e27fc640 1 -- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 msgr2=0x7f59e40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e27fc640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e27fc640 1 -- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59e40cd390 msgr2=0x7f59e414a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e27fc640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59e40cd390 0x7f59e414a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e27fc640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e4156d80 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e37fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59e415aeb0 0x7f59e40b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e2ffd640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59e40cd390 0x7f59e414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e27fc640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40b06f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f59d800d800 tx=0x7f59d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d800b6c0 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d8019070 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e4157060 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d800ba80 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.617+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e4157620 con 0x7f59e40ceba0 2024-03-08T08:31:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.618+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59e40749c0 con 0x7f59e40ceba0 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.618+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59d800bc20 con 0x7f59e40ceba0 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59cb7fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f59d808a3b0 con 0x7f59e40ceba0 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59e2ffd640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59cb7fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7f59d00713c0 0x7f59d00737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59d8017e50 con 0x7f59d00713c0 2024-03-08T08:31:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59d8091050 con 0x7f59e40ceba0 2024-03-08T08:31:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59e37fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7f59d00713c0 0x7f59d00737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59e2ffd640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0015e0 tx=0x7f59dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.619+0000 7f59e37fe640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7f59d00713c0 0x7f59d00737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.620+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== osd.0 v2:172.21.15.12:6809/1155150761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f59d8017e50 con 0x7f59d00713c0 2024-03-08T08:31:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.633+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 --> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f59e40b3000 con 0x7f59d00713c0 2024-03-08T08:31:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59cb7fe640 1 -- 172.21.15.12:0/1829428494 <== osd.0 v2:172.21.15.12:6809/1155150761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f59e40b3000 con 0x7f59d00713c0 2024-03-08T08:31:41.635 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:41.635 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:41.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7f59d00713c0 msgr2=0x7f59d00737e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6809/1155150761,v1:172.21.15.12:6812/1155150761] conn(0x7f59d00713c0 0x7f59d00737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f59d006b500 msgr2=0x7f59d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0015e0 tx=0x7f59dc009040 comp rx=0 tx=0).stop 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 msgr2=0x7f59e40b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 --2- 172.21.15.12:0/1829428494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59e40ceba0 0x7f59e40b06f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f59d800d800 tx=0x7f59d800dcc0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e37fe640 1 -- 172.21.15.12:0/1829428494 reap_dead start 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 shutdown_connections 2024-03-08T08:31:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 >> 172.21.15.12:0/1829428494 conn(0x7f59e406abd0 msgr2=0x7f59e415a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 shutdown_connections 2024-03-08T08:31:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.634+0000 7f59e9519640 1 -- 172.21.15.12:0/1829428494 wait complete. 2024-03-08T08:31:41.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-08T08:31:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.747+0000 7eff5e60e640 1 Processor -- start 2024-03-08T08:31:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.747+0000 7eff5e60e640 1 -- start start 2024-03-08T08:31:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff580cfa00 0x7eff580cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 0x7eff580cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff580d0320 0x7eff580b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5809a4f0 con 0x7eff580d0320 2024-03-08T08:31:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580cd820 con 0x7eff5809a980 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5e60e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff58099bc0 con 0x7eff580cfa00 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5cbab640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff580d0320 0x7eff580b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5cbab640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff580d0320 0x7eff580b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54550/0 (socket says 172.21.15.12:54550) 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.748+0000 7eff5cbab640 1 -- 172.21.15.12:0/2140814153 learned_addr learned my addr 172.21.15.12:0/2140814153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff577fe640 1 --2- 172.21.15.12:0/2140814153 >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 0x7eff580cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff57fff640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff580cfa00 0x7eff580cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 -- 172.21.15.12:0/2140814153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff580cfa00 msgr2=0x7eff580cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff580cfa00 0x7eff580cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 -- 172.21.15.12:0/2140814153 >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 msgr2=0x7eff580cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 --2- 172.21.15.12:0/2140814153 >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 0x7eff580cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 -- 172.21.15.12:0/2140814153 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58099ea0 con 0x7eff580d0320 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff577fe640 1 --2- 172.21.15.12:0/2140814153 >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 0x7eff580cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.749+0000 7eff5cbab640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580b12e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7eff58072930 tx=0x7eff5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f881499dfe27d2a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff56ffd640 1 -- 172.21.15.12:0/2140814153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5002ca70 con 0x7eff580d0320 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff56ffd640 1 -- 172.21.15.12:0/2140814153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5002cc10 con 0x7eff580d0320 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff56ffd640 1 -- 172.21.15.12:0/2140814153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff500343f0 con 0x7eff580d0320 2024-03-08T08:31:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 -- 172.21.15.12:0/2140814153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 msgr2=0x7eff580b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580b12e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7eff58072930 tx=0x7eff5002bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 -- 172.21.15.12:0/2140814153 shutdown_connections 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580b12e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 --2- 172.21.15.12:0/2140814153 >> v2:172.21.15.156:3300/0 conn(0x7eff5809a980 0x7eff580cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 --2- 172.21.15.12:0/2140814153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff580cfa00 0x7eff580cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 -- 172.21.15.12:0/2140814153 >> 172.21.15.12:0/2140814153 conn(0x7eff580b3470 msgr2=0x7eff580b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 -- 172.21.15.12:0/2140814153 shutdown_connections 2024-03-08T08:31:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.750+0000 7eff5e60e640 1 -- 172.21.15.12:0/2140814153 wait complete. 2024-03-08T08:31:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.751+0000 7eff5e60e640 1 Processor -- start 2024-03-08T08:31:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.751+0000 7eff5e60e640 1 -- start start 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 0x7eff5815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff580cfa00 0x7eff580c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff580cd480 con 0x7eff580d0320 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5809ade0 con 0x7eff580cfa00 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5e60e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff580cd600 con 0x7eff5809a980 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5cbab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5cbab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54560/0 (socket says 172.21.15.12:54560) 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.752+0000 7eff5cbab640 1 -- 172.21.15.12:0/1128703694 learned_addr learned my addr 172.21.15.12:0/1128703694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff577fe640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff580cfa00 0x7eff580c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff57fff640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 0x7eff5815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 -- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 msgr2=0x7eff5815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 0x7eff5815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 -- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff580cfa00 msgr2=0x7eff580c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff580cfa00 0x7eff580c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 -- 172.21.15.12:0/1128703694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580c92a0 con 0x7eff580d0320 2024-03-08T08:31:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff57fff640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 0x7eff5815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5cbab640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff50036040 tx=0x7eff50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50044020 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50040670 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff580c9580 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50009510 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.753+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5815f430 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.754+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff580749c0 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.754+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff50047020 con 0x7eff580d0320 2024-03-08T08:31:41.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.755+0000 7eff54ff9640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 0x7eff3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:41.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.755+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7eff500ac9e0 con 0x7eff580d0320 2024-03-08T08:31:41.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.755+0000 7eff57fff640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 0x7eff3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:41.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.755+0000 7eff57fff640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 0x7eff3806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b820 tx=0x7eff48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:41.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.759+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7eff5004e030 con 0x7eff580d0320 2024-03-08T08:31:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.941+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7eff580ce170 con 0x7eff3806b4d0 2024-03-08T08:31:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.941+0000 7eff54ff9640 1 -- 172.21.15.12:0/1128703694 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7eff580ce170 con 0x7eff3806b4d0 2024-03-08T08:31:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2024-03-08T08:31:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 msgr2=0x7eff3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 0x7eff3806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b820 tx=0x7eff48008040 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 msgr2=0x7eff580c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff50036040 tx=0x7eff50004a40 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 shutdown_connections 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7eff3806b4d0 0x7eff3806d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff580d0320 0x7eff580c6500 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff580cfa00 0x7eff580c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 --2- 172.21.15.12:0/1128703694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff5809a980 0x7eff5815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 >> 172.21.15.12:0/1128703694 conn(0x7eff580b3470 msgr2=0x7eff580ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 shutdown_connections 2024-03-08T08:31:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:41.943+0000 7eff5e60e640 1 -- 172.21.15.12:0/1128703694 wait complete. 2024-03-08T08:31:41.955 INFO:tasks.ceph:Scrubbing osd.1 2024-03-08T08:31:41.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:42.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.060+0000 7f5a5b046640 1 Processor -- start 2024-03-08T08:31:42.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.060+0000 7f5a5b046640 1 -- start start 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a5b046640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a5415e050 0x7f5a540d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a5b046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a5b046640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540ce580 con 0x7f5a541623d0 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a5b046640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540ce860 con 0x7f5a5415e050 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a5b046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540ceb40 con 0x7f5a540b29a0 2024-03-08T08:31:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a5415e050 0x7f5a540d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a53fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5a541623d0 0x7f5a540b5600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37016/0 (socket says 172.21.15.12:37016) 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a53fff640 1 -- 172.21.15.12:0/2887435228 learned_addr learned my addr 172.21.15.12:0/2887435228 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a5415e050 0x7f5a540d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54568/0 (socket says 172.21.15.12:54568) 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a595e3640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 -- 172.21.15.12:0/2887435228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 msgr2=0x7f5a540d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 -- 172.21.15.12:0/2887435228 >> v1:172.21.15.12:6789/0 conn(0x7f5a541623d0 legacy=0x7f5a540b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a58de2640 1 -- 172.21.15.12:0/2887435228 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540cf340 con 0x7f5a5415e050 2024-03-08T08:31:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.061+0000 7f5a537fe640 1 -- 172.21.15.12:0/2887435228 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121841847 0 0) 0x7f5a540ce580 con 0x7f5a541623d0 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.062+0000 7f5a58de2640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a5415e050 0x7f5a540d0ae0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009d90 tx=0x7f5a4c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8b8f99b6d4c467 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.062+0000 7f5a537fe640 1 -- 172.21.15.12:0/2887435228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c039070 con 0x7f5a5415e050 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.062+0000 7f5a537fe640 1 -- 172.21.15.12:0/2887435228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4c02cbc0 con 0x7f5a5415e050 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.062+0000 7f5a537fe640 1 -- 172.21.15.12:0/2887435228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c034440 con 0x7f5a5415e050 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 -- 172.21.15.12:0/2887435228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a5415e050 msgr2=0x7f5a540d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a5415e050 0x7f5a540d0ae0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009d90 tx=0x7f5a4c02be10 comp rx=0 tx=0).stop 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 -- 172.21.15.12:0/2887435228 shutdown_connections 2024-03-08T08:31:42.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 --2- 172.21.15.12:0/2887435228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a5415e050 0x7f5a540d0ae0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 -- 172.21.15.12:0/2887435228 >> 172.21.15.12:0/2887435228 conn(0x7f5a5406c7c0 msgr2=0x7f5a5406cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 -- 172.21.15.12:0/2887435228 shutdown_connections 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.063+0000 7f5a5b046640 1 -- 172.21.15.12:0/2887435228 wait complete. 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.064+0000 7f5a5b046640 1 Processor -- start 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.064+0000 7f5a5b046640 1 -- start start 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a5415e050 0x7f5a541526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 0x7f5a540b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540b3200 con 0x7f5a541623d0 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540b3080 con 0x7f5a5415e050 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a5b046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540b5d10 con 0x7f5a540b29a0 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 0x7f5a540b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 0x7f5a540b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54570/0 (socket says 172.21.15.12:54570) 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 -- 172.21.15.12:0/936623696 learned_addr learned my addr 172.21.15.12:0/936623696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a53fff640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a5415e050 0x7f5a541526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 -- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 msgr2=0x7f5a540c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a540b29a0 0x7f5a540c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 -- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a5415e050 msgr2=0x7f5a541526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a5415e050 0x7f5a541526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540b6de0 con 0x7f5a541623d0 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a53fff640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a5415e050 0x7f5a541526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.065+0000 7f5a595e3640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 0x7f5a540b8790 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400d800 tx=0x7f5a4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400b6c0 con 0x7f5a541623d0 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a44019070 con 0x7f5a541623d0 2024-03-08T08:31:42.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a540b70c0 con 0x7f5a541623d0 2024-03-08T08:31:42.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400bac0 con 0x7f5a541623d0 2024-03-08T08:31:42.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a540b7680 con 0x7f5a541623d0 2024-03-08T08:31:42.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.066+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a540749c0 con 0x7f5a541623d0 2024-03-08T08:31:42.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.067+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a4400bc60 con 0x7f5a541623d0 2024-03-08T08:31:42.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a517fa640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a4006b4d0 0x7f5a4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5a4408ae50 con 0x7f5a541623d0 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a58de2640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a4006b4d0 0x7f5a4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a517fa640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7f5a40071390 0x7f5a400737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a40073e60 con 0x7f5a40071390 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a4408b2e0 con 0x7f5a541623d0 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a53fff640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7f5a40071390 0x7f5a400737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a58de2640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a4006b4d0 0x7f5a4006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c036000 tx=0x7f5a4c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.068+0000 7f5a53fff640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7f5a40071390 0x7f5a400737b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.069+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== osd.1 v2:172.21.15.12:6808/1692589129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5a40073e60 con 0x7f5a40071390 2024-03-08T08:31:42.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.082+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 --> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5a540b9510 con 0x7f5a40071390 2024-03-08T08:31:42.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.082+0000 7f5a517fa640 1 -- 172.21.15.12:0/936623696 <== osd.1 v2:172.21.15.12:6808/1692589129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5a540b9510 con 0x7f5a40071390 2024-03-08T08:31:42.084 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:42.084 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:42.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.082+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7f5a40071390 msgr2=0x7f5a400737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.082+0000 7f5a5b046640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6808/1692589129,v1:172.21.15.12:6811/1692589129] conn(0x7f5a40071390 0x7f5a400737b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a4006b4d0 msgr2=0x7f5a4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5a4006b4d0 0x7f5a4006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c036000 tx=0x7f5a4c02e000 comp rx=0 tx=0).stop 2024-03-08T08:31:42.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 msgr2=0x7f5a540b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 --2- 172.21.15.12:0/936623696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a541623d0 0x7f5a540b8790 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400d800 tx=0x7f5a4400dcc0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a595e3640 1 -- 172.21.15.12:0/936623696 reap_dead start 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 shutdown_connections 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 >> 172.21.15.12:0/936623696 conn(0x7f5a5406c7c0 msgr2=0x7f5a5406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 shutdown_connections 2024-03-08T08:31:42.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.083+0000 7f5a5b046640 1 -- 172.21.15.12:0/936623696 wait complete. 2024-03-08T08:31:42.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-08T08:31:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.195+0000 7f862ade2640 1 Processor -- start 2024-03-08T08:31:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.195+0000 7f862ade2640 1 -- start start 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862ade2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d1110 0x7f86240b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862ade2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f86240d31c0 0x7f86240b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862ade2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862415e1a0 con 0x7f862415ddc0 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862ade2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86240b9030 con 0x7f86240d31c0 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862ade2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240d1d20 con 0x7f86240d1110 2024-03-08T08:31:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862937f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f86240d31c0 0x7f86240b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f8623fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d1110 0x7f86240b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862937f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f86240d31c0 0x7f86240b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:49694/0 (socket says 172.21.15.12:49694) 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f8628b7e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f862415ddc0 0x7f86240d0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:36516/0 (socket says 172.21.15.12:36516) 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.196+0000 7f862937f640 1 -- 172.21.15.12:0/3216878808 learned_addr learned my addr 172.21.15.12:0/3216878808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.197+0000 7f862937f640 1 -- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d1110 msgr2=0x7f86240b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.197+0000 7f862937f640 1 --2- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d1110 0x7f86240b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.197+0000 7f862937f640 1 -- 172.21.15.12:0/3216878808 >> v1:172.21.15.156:6789/0 conn(0x7f862415ddc0 legacy=0x7f86240d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.197+0000 7f862937f640 1 -- 172.21.15.12:0/3216878808 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86240b8790 con 0x7f86240d31c0 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.197+0000 7f862937f640 1 --2- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86240d31c0 0x7f86240b7120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f861c009e50 tx=0x7f861c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b626348c33f6bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f86237fe640 1 -- 172.21.15.12:0/3216878808 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c02cb00 con 0x7f86240d31c0 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f86237fe640 1 -- 172.21.15.12:0/3216878808 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c02cca0 con 0x7f86240d31c0 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f86237fe640 1 -- 172.21.15.12:0/3216878808 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c034610 con 0x7f86240d31c0 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 -- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86240d31c0 msgr2=0x7f86240b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 --2- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86240d31c0 0x7f86240b7120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f861c009e50 tx=0x7f861c02bf60 comp rx=0 tx=0).stop 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 -- 172.21.15.12:0/3216878808 shutdown_connections 2024-03-08T08:31:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 --2- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86240d31c0 0x7f86240b7120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 --2- 172.21.15.12:0/3216878808 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d1110 0x7f86240b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 -- 172.21.15.12:0/3216878808 >> 172.21.15.12:0/3216878808 conn(0x7f8624072720 msgr2=0x7f8624072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 -- 172.21.15.12:0/3216878808 shutdown_connections 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.198+0000 7f862ade2640 1 -- 172.21.15.12:0/3216878808 wait complete. 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.199+0000 7f862ade2640 1 Processor -- start 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.199+0000 7f862ade2640 1 -- start start 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d31c0 0x7f86241f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862415ddc0 0x7f8624160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b9980 con 0x7f86240d1110 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b8d50 con 0x7f862415ddc0 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862ade2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b7660 con 0x7f86240d31c0 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f8628b7e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f8628b7e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54584/0 (socket says 172.21.15.12:54584) 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f8628b7e640 1 -- 172.21.15.12:0/1342523480 learned_addr learned my addr 172.21.15.12:0/1342523480 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f8623fff640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d31c0 0x7f86241f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.200+0000 7f862937f640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862415ddc0 0x7f8624160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 -- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d31c0 msgr2=0x7f86241f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d31c0 0x7f86241f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 -- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862415ddc0 msgr2=0x7f8624160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862415ddc0 0x7f8624160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 -- 172.21.15.12:0/1342523480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86240aa4b0 con 0x7f86240d1110 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f8628b7e640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f861800c3c0 tx=0x7f861800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86180074e0 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8618007680 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86240aa790 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618004a00 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.201+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86240aad50 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.202+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86240749c0 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.202+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f861801a020 con 0x7f86240d1110 2024-03-08T08:31:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.203+0000 7f86217fa640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 0x7f860806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.203+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8618089490 con 0x7f86240d1110 2024-03-08T08:31:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.203+0000 7f8623fff640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 0x7f860806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.203+0000 7f8623fff640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 0x7f860806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86241f8080 tx=0x7f8610009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.207+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8618058050 con 0x7f86240d1110 2024-03-08T08:31:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.388+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f86240b3dd0 con 0x7f860806b4d0 2024-03-08T08:31:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.389+0000 7f86217fa640 1 -- 172.21.15.12:0/1342523480 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f86240b3dd0 con 0x7f860806b4d0 2024-03-08T08:31:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2024-03-08T08:31:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.390+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 msgr2=0x7f860806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.390+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 0x7f860806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86241f8080 tx=0x7f8610009190 comp rx=0 tx=0).stop 2024-03-08T08:31:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.390+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 msgr2=0x7f862415c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.390+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f861800c3c0 tx=0x7f861800c880 comp rx=0 tx=0).stop 2024-03-08T08:31:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.390+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 shutdown_connections 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f860806b4d0 0x7f860806d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f862415ddc0 0x7f8624160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86240d31c0 0x7f86241f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 --2- 172.21.15.12:0/1342523480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86240d1110 0x7f862415c020 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 >> 172.21.15.12:0/1342523480 conn(0x7f8624072720 msgr2=0x7f86240b7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 shutdown_connections 2024-03-08T08:31:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.391+0000 7f862ade2640 1 -- 172.21.15.12:0/1342523480 wait complete. 2024-03-08T08:31:42.403 INFO:tasks.ceph:Scrubbing osd.2 2024-03-08T08:31:42.403 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.507+0000 7f5b957ac640 1 Processor -- start 2024-03-08T08:31:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.507+0000 7f5b957ac640 1 -- start start 2024-03-08T08:31:42.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b957ac640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b900d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b957ac640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900cfaa0 con 0x7f5b9015f200 2024-03-08T08:31:42.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b957ac640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900b1390 con 0x7f5b900b5f10 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b957ac640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900b1160 con 0x7f5b90067230 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b8effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5b9015f200 0x7f5b90066b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37020/0 (socket says 172.21.15.12:37020) 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b8effd640 1 -- 172.21.15.12:0/2555398986 learned_addr learned my addr 172.21.15.12:0/2555398986 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b900d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668212590 0 0) 0x7f5b900cfaa0 con 0x7f5b9015f200 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.508+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c003520 con 0x7f5b9015f200 2024-03-08T08:31:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749711660 0 0) 0x7f5b900b1390 con 0x7f5b900b5f10 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b900cfaa0 con 0x7f5b900b5f10 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664363113 0 0) 0x7f5b7c003520 con 0x7f5b9015f200 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 msgr2=0x7f5b900d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b900d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 >> v1:172.21.15.156:6789/0 conn(0x7f5b900b5f10 legacy=0x7f5b900d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900b1670 con 0x7f5b9015f200 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f5b88001670 con 0x7f5b9015f200 2024-03-08T08:31:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 >> v1:172.21.15.12:6789/0 conn(0x7f5b9015f200 legacy=0x7f5b90066b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c003b70 0x7f5b7c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b7c004800 0x7f5b7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c00ba70 con 0x7f5b7c004800 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c00bbf0 con 0x7f5b7c003b70 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c00bdf0 con 0x7f5b90067230 2024-03-08T08:31:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b7c004800 0x7f5b7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8f7fe640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8effd640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c003b70 0x7f5b7c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 -- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 msgr2=0x7f5b7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b7c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 -- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c003b70 msgr2=0x7f5b7c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c003b70 0x7f5b7c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8e7fc640 1 -- 172.21.15.12:0/2555398986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c003520 con 0x7f5b7c004800 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8f7fe640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b90067230 0x7f5b7c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.509+0000 7f5b8effd640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c003b70 0x7f5b7c003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b7c004800 0x7f5b7c004c80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009a90 tx=0x7f5b7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64356e58ceeade1d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78039070 con 0x7f5b7c004800 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7802c680 con 0x7f5b7c004800 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b8dffb640 1 -- 172.21.15.12:0/2555398986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78009690 con 0x7f5b7c004800 2024-03-08T08:31:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b957ac640 1 -- 172.21.15.12:0/2555398986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b7c004800 msgr2=0x7f5b7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b957ac640 1 --2- 172.21.15.12:0/2555398986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b7c004800 0x7f5b7c004c80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009a90 tx=0x7f5b7c005830 comp rx=0 tx=0).stop 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.510+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/2555398986 reap_dead start 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.511+0000 7f5b957ac640 1 -- 172.21.15.12:0/2555398986 shutdown_connections 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.511+0000 7f5b957ac640 1 -- 172.21.15.12:0/2555398986 >> 172.21.15.12:0/2555398986 conn(0x7f5b90099a40 msgr2=0x7f5b900b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.511+0000 7f5b957ac640 1 -- 172.21.15.12:0/2555398986 shutdown_connections 2024-03-08T08:31:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.511+0000 7f5b957ac640 1 -- 172.21.15.12:0/2555398986 wait complete. 2024-03-08T08:31:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.512+0000 7f5b957ac640 1 Processor -- start 2024-03-08T08:31:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.512+0000 7f5b957ac640 1 -- start start 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c004800 0x7f5b900aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b900ae280 0x7f5b9015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 0x7f5b90146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900cf850 con 0x7f5b901465b0 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900d0c80 con 0x7f5b7c004800 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900d0e00 con 0x7f5b900ae280 2024-03-08T08:31:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 0x7f5b90146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 0x7f5b90146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54606/0 (socket says 172.21.15.12:54606) 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/1807607122 learned_addr learned my addr 172.21.15.12:0/1807607122 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8effd640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c004800 0x7f5b900aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b900ae280 0x7f5b9015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b900ae280 msgr2=0x7f5b9015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b900ae280 0x7f5b9015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c004800 msgr2=0x7f5b900aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b7c004800 0x7f5b900aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b901f6f50 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5b900ae280 0x7f5b9015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b8f7fe640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 0x7f5b90146a10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5b80001330 tx=0x7f5b8000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000ee50 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b80003b80 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b80003d70 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b900ae6e0 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.513+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b901f74d0 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.514+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b90067670 con 0x7f5b901465b0 2024-03-08T08:31:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b8001a020 con 0x7f5b901465b0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5b7006b500 0x7f5b7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5b80089340 con 0x7f5b901465b0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b8effd640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5b7006b500 0x7f5b7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f5b700713c0 0x7f5b700737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b80016ec0 con 0x7f5b700713c0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b80090050 con 0x7f5b901465b0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.515+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f5b700713c0 0x7f5b700737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.516+0000 7f5b8effd640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5b7006b500 0x7f5b7006d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5b88000c00 tx=0x7f5b88007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.516+0000 7f5b8e7fc640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f5b700713c0 0x7f5b700737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.516+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== osd.2 v2:172.21.15.12:6800/2701699187 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5b80016ec0 con 0x7f5b700713c0 2024-03-08T08:31:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 --> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5b900749c0 con 0x7f5b700713c0 2024-03-08T08:31:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b877fe640 1 -- 172.21.15.12:0/1807607122 <== osd.2 v2:172.21.15.12:6800/2701699187 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5b900749c0 con 0x7f5b700713c0 2024-03-08T08:31:42.531 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:42.531 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:42.532 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f5b700713c0 msgr2=0x7f5b700737e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6800/2701699187,v1:172.21.15.12:6801/2701699187] conn(0x7f5b700713c0 0x7f5b700737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5b7006b500 msgr2=0x7f5b7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f5b7006b500 0x7f5b7006d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5b88000c00 tx=0x7f5b88007040 comp rx=0 tx=0).stop 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 msgr2=0x7f5b90146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 --2- 172.21.15.12:0/1807607122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b901465b0 0x7f5b90146a10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5b80001330 tx=0x7f5b8000ec00 comp rx=0 tx=0).stop 2024-03-08T08:31:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b8f7fe640 1 -- 172.21.15.12:0/1807607122 reap_dead start 2024-03-08T08:31:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 shutdown_connections 2024-03-08T08:31:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 >> 172.21.15.12:0/1807607122 conn(0x7f5b90099a40 msgr2=0x7f5b9006f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 shutdown_connections 2024-03-08T08:31:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.530+0000 7f5b957ac640 1 -- 172.21.15.12:0/1807607122 wait complete. 2024-03-08T08:31:42.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-08T08:31:42.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.642+0000 7f859d0d4640 1 Processor -- start 2024-03-08T08:31:42.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.642+0000 7f859d0d4640 1 -- start start 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859d0d4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f859815dbc0 0x7f85980d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859d0d4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980b7540 con 0x7f85980b9760 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859d0d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980b7820 con 0x7f85980d2050 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859d0d4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980b8150 con 0x7f859815dbc0 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f8596d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f859815dbc0 0x7f85980d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859759e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f85980b9760 0x7f85980b85e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37024/0 (socket says 172.21.15.12:37024) 2024-03-08T08:31:42.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f859759e640 1 -- 172.21.15.12:0/1830038964 learned_addr learned my addr 172.21.15.12:0/1830038964 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:42.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f8596d9d640 1 -- 172.21.15.12:0/1830038964 >> v1:172.21.15.156:6789/0 conn(0x7f85980d2050 legacy=0x7f85980b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f8596d9d640 1 -- 172.21.15.12:0/1830038964 >> v1:172.21.15.12:6789/0 conn(0x7f85980b9760 legacy=0x7f85980b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.643+0000 7f8596d9d640 1 -- 172.21.15.12:0/1830038964 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980b7b00 con 0x7f859815dbc0 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.644+0000 7f8596d9d640 1 --2- 172.21.15.12:0/1830038964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f859815dbc0 0x7f85980d1b10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f85880097a0 tx=0x7f858802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97de30ad992541f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.644+0000 7f8595d9b640 1 -- 172.21.15.12:0/1830038964 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588034070 con 0x7f859815dbc0 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.644+0000 7f8595d9b640 1 -- 172.21.15.12:0/1830038964 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8588030610 con 0x7f859815dbc0 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.644+0000 7f8595d9b640 1 -- 172.21.15.12:0/1830038964 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880309e0 con 0x7f859815dbc0 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 -- 172.21.15.12:0/1830038964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f859815dbc0 msgr2=0x7f85980d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1830038964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f859815dbc0 0x7f85980d1b10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f85880097a0 tx=0x7f858802b960 comp rx=0 tx=0).stop 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 -- 172.21.15.12:0/1830038964 shutdown_connections 2024-03-08T08:31:42.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1830038964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f859815dbc0 0x7f85980d1b10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 -- 172.21.15.12:0/1830038964 >> 172.21.15.12:0/1830038964 conn(0x7f85980726f0 msgr2=0x7f8598072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 -- 172.21.15.12:0/1830038964 shutdown_connections 2024-03-08T08:31:42.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.645+0000 7f859d0d4640 1 -- 172.21.15.12:0/1830038964 wait complete. 2024-03-08T08:31:42.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.646+0000 7f859d0d4640 1 Processor -- start 2024-03-08T08:31:42.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.646+0000 7f859d0d4640 1 -- start start 2024-03-08T08:31:42.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f859815dbc0 0x7f8598152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8598153430 0x7f85980c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815dfa0 con 0x7f85980b9760 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980b7250 con 0x7f8598153430 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f859d0d4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980b8cf0 con 0x7f859815dbc0 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54632/0 (socket says 172.21.15.12:54632) 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 -- 172.21.15.12:0/1098382427 learned_addr learned my addr 172.21.15.12:0/1098382427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 -- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f859815dbc0 msgr2=0x7f8598152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f859815dbc0 0x7f8598152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 -- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8598153430 msgr2=0x7f85980c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8598153430 0x7f85980c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.647+0000 7f8596d9d640 1 -- 172.21.15.12:0/1098382427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598154fa0 con 0x7f85980b9760 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f8596d9d640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f85880098c0 tx=0x7f85880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588034040 con 0x7f85980b9760 2024-03-08T08:31:42.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858803acc0 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588044630 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85980c81a0 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.648+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85980c7de0 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.649+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85980749c0 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.649+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f858804b060 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.650+0000 7f857f7fe640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 0x7f858406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.650+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f85880abeb0 con 0x7f85980b9760 2024-03-08T08:31:42.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.650+0000 7f859659c640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 0x7f858406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.650+0000 7f859659c640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 0x7f858406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f85981538f0 tx=0x7f858c007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.654+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f858803f070 con 0x7f85980b9760 2024-03-08T08:31:42.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.833+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f85980d3060 con 0x7f858406b4d0 2024-03-08T08:31:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.834+0000 7f857f7fe640 1 -- 172.21.15.12:0/1098382427 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f85980d3060 con 0x7f858406b4d0 2024-03-08T08:31:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2024-03-08T08:31:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.835+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 msgr2=0x7f858406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.835+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 0x7f858406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f85981538f0 tx=0x7f858c007990 comp rx=0 tx=0).stop 2024-03-08T08:31:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.835+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 msgr2=0x7f85980caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.835+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f85880098c0 tx=0x7f85880046c0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 shutdown_connections 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f858406b4d0 0x7f858406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8598153430 0x7f85980c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f859815dbc0 0x7f8598152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 --2- 172.21.15.12:0/1098382427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85980b9760 0x7f85980caba0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 >> 172.21.15.12:0/1098382427 conn(0x7f85980726f0 msgr2=0x7f85980b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 shutdown_connections 2024-03-08T08:31:42.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.836+0000 7f859d0d4640 1 -- 172.21.15.12:0/1098382427 wait complete. 2024-03-08T08:31:42.848 INFO:tasks.ceph:Scrubbing osd.3 2024-03-08T08:31:42.848 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:42.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.952+0000 7f32ff6ae640 1 Processor -- start 2024-03-08T08:31:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.952+0000 7f32ff6ae640 1 -- start start 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f815db10 0x7f32f80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f32f80ccc10 0x7f32f8099c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32f809a140 0x7f32f8161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80cdd30 con 0x7f32f809a140 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80cd1b0 con 0x7f32f80ccc10 2024-03-08T08:31:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32ff6ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80cd490 con 0x7f32f815db10 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32f809a140 0x7f32f8161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32f809a140 0x7f32f8161800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54642/0 (socket says 172.21.15.12:54642) 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 -- 172.21.15.12:0/375979077 learned_addr learned my addr 172.21.15.12:0/375979077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fd44a640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f815db10 0x7f32f80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fcc49640 1 --2- 172.21.15.12:0/375979077 >> v2:172.21.15.156:3300/0 conn(0x7f32f80ccc10 0x7f32f8099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 -- 172.21.15.12:0/375979077 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f815db10 msgr2=0x7f32f80cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f815db10 0x7f32f80cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 -- 172.21.15.12:0/375979077 >> v2:172.21.15.156:3300/0 conn(0x7f32f80ccc10 msgr2=0x7f32f8099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/375979077 >> v2:172.21.15.156:3300/0 conn(0x7f32f80ccc10 0x7f32f8099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.953+0000 7f32fdc4b640 1 -- 172.21.15.12:0/375979077 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80cd770 con 0x7f32f809a140 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.954+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f809a140 0x7f32f8161800 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f32f8072bd0 tx=0x7f32f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc4dd5c84301970 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.954+0000 7f32effff640 1 -- 172.21.15.12:0/375979077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f402ca70 con 0x7f32f809a140 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.954+0000 7f32effff640 1 -- 172.21.15.12:0/375979077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f402cc10 con 0x7f32f809a140 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.954+0000 7f32effff640 1 -- 172.21.15.12:0/375979077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f40343f0 con 0x7f32f809a140 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 -- 172.21.15.12:0/375979077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f809a140 msgr2=0x7f32f8161800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f809a140 0x7f32f8161800 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f32f8072bd0 tx=0x7f32f402bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 -- 172.21.15.12:0/375979077 shutdown_connections 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f809a140 0x7f32f8161800 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/375979077 >> v2:172.21.15.156:3300/0 conn(0x7f32f80ccc10 0x7f32f8099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/375979077 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f815db10 0x7f32f80cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 -- 172.21.15.12:0/375979077 >> 172.21.15.12:0/375979077 conn(0x7f32f80b33d0 msgr2=0x7f32f80b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 -- 172.21.15.12:0/375979077 shutdown_connections 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.955+0000 7f32ff6ae640 1 -- 172.21.15.12:0/375979077 wait complete. 2024-03-08T08:31:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.956+0000 7f32ff6ae640 1 Processor -- start 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.956+0000 7f32ff6ae640 1 -- start start 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f809a140 0x7f32f806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 0x7f32f806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32f815db10 0x7f32f80ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f815e070 con 0x7f32f80ccc10 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f815def0 con 0x7f32f815db10 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32ff6ae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8161d40 con 0x7f32f809a140 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 0x7f32f806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 0x7f32f806cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54658/0 (socket says 172.21.15.12:54658) 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 -- 172.21.15.12:0/2226299949 learned_addr learned my addr 172.21.15.12:0/2226299949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32f815db10 0x7f32f80ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fd44a640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f809a140 0x7f32f806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 -- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f809a140 msgr2=0x7f32f806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f32f809a140 0x7f32f806ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 -- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32f815db10 msgr2=0x7f32f80ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32f815db10 0x7f32f80ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fcc49640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80cb680 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.957+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32f815db10 0x7f32f80ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32fcc49640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 0x7f32f806cfb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d800 tx=0x7f32e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8018020 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e80096a0 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80b4940 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8009a70 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.958+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80b4f00 con 0x7f32f80ccc10 2024-03-08T08:31:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.959+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32f80b4730 con 0x7f32f80ccc10 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.959+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32e801b020 con 0x7f32f80ccc10 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32edffb640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f32dc06b4d0 0x7f32dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f32e808a210 con 0x7f32f80ccc10 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32fd44a640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f32dc06b4d0 0x7f32dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32edffb640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f32dc071390 0x7f32dc0737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32e800cfb0 con 0x7f32dc071390 2024-03-08T08:31:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e8091050 con 0x7f32f80ccc10 2024-03-08T08:31:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f32dc071390 0x7f32dc0737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32fd44a640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f32dc06b4d0 0x7f32dc06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32f0006fd0 tx=0x7f32f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.960+0000 7f32fdc4b640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f32dc071390 0x7f32dc0737b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.961+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== osd.3 v2:172.21.15.12:6810/2394545747 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f32e800cfb0 con 0x7f32dc071390 2024-03-08T08:31:42.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.974+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 --> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f32f8070330 con 0x7f32dc071390 2024-03-08T08:31:42.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.974+0000 7f32edffb640 1 -- 172.21.15.12:0/2226299949 <== osd.3 v2:172.21.15.12:6810/2394545747 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f32f8070330 con 0x7f32dc071390 2024-03-08T08:31:42.975 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:42.976 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:42.976 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.974+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f32dc071390 msgr2=0x7f32dc0737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.974+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6810/2394545747,v1:172.21.15.12:6813/2394545747] conn(0x7f32dc071390 0x7f32dc0737b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f32dc06b4d0 msgr2=0x7f32dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f32dc06b4d0 0x7f32dc06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32f0006fd0 tx=0x7f32f0008040 comp rx=0 tx=0).stop 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 msgr2=0x7f32f806cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:42.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 --2- 172.21.15.12:0/2226299949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f80ccc10 0x7f32f806cfb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d800 tx=0x7f32e800dcc0 comp rx=0 tx=0).stop 2024-03-08T08:31:42.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32fdc4b640 1 -- 172.21.15.12:0/2226299949 reap_dead start 2024-03-08T08:31:42.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 shutdown_connections 2024-03-08T08:31:42.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 >> 172.21.15.12:0/2226299949 conn(0x7f32f80b33d0 msgr2=0x7f32f815ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:42.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 shutdown_connections 2024-03-08T08:31:42.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:42.975+0000 7f32ff6ae640 1 -- 172.21.15.12:0/2226299949 wait complete. 2024-03-08T08:31:42.986 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-08T08:31:43.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.087+0000 7fcec1e4e640 1 Processor -- start 2024-03-08T08:31:43.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 -- start start 2024-03-08T08:31:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcebc0b9760 0x7fcebc0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc0b7080 0x7fcebc0b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0d2220 con 0x7fcebc15dbc0 2024-03-08T08:31:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc0b8150 con 0x7fcebc0b9760 2024-03-08T08:31:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcec1e4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b7540 con 0x7fcebc0b7080 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcebaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcebc0b9760 0x7fcebc0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcebb7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcebc15dbc0 0x7fcebc0d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37038/0 (socket says 172.21.15.12:37038) 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcebb7fe640 1 -- 172.21.15.12:0/1791406704 learned_addr learned my addr 172.21.15.12:0/1791406704 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.088+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1791406704 >> v2:172.21.15.12:3300/0 conn(0x7fcebc0b9760 0x7fcebc0b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54666/0 (socket says 172.21.15.12:54666) 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fcebaffd640 1 -- 172.21.15.12:0/1791406704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc0b7080 msgr2=0x7fcebc0b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fcebbfff640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc0b7080 0x7fcebc0b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc0b7080 0x7fcebc0b85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fcebaffd640 1 -- 172.21.15.12:0/1791406704 >> v1:172.21.15.12:6789/0 conn(0x7fcebc15dbc0 legacy=0x7fcebc0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fcebaffd640 1 -- 172.21.15.12:0/1791406704 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0b7820 con 0x7fcebc0b9760 2024-03-08T08:31:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.089+0000 7fceba7fc640 1 -- 172.21.15.12:0/1791406704 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559514420 0 0) 0x7fcebc0d2220 con 0x7fcebc15dbc0 2024-03-08T08:31:43.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b9760 0x7fcebc0b6b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fceb00097a0 tx=0x7fceb002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e299c78631d4ef server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fceba7fc640 1 -- 172.21.15.12:0/1791406704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0034070 con 0x7fcebc0b9760 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fceba7fc640 1 -- 172.21.15.12:0/1791406704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb0030610 con 0x7fcebc0b9760 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fceba7fc640 1 -- 172.21.15.12:0/1791406704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00309e0 con 0x7fcebc0b9760 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1791406704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b9760 msgr2=0x7fcebc0b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b9760 0x7fcebc0b6b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fceb00097a0 tx=0x7fceb002b960 comp rx=0 tx=0).stop 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1791406704 shutdown_connections 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc0b7080 0x7fcebc0b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1791406704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b9760 0x7fcebc0b6b40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1791406704 >> 172.21.15.12:0/1791406704 conn(0x7fcebc0726f0 msgr2=0x7fcebc072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1791406704 shutdown_connections 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.090+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1791406704 wait complete. 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.091+0000 7fcec1e4e640 1 Processor -- start 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 -- start start 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 0x7fcebc160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 0x7fcebc15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc15dfa0 con 0x7fcebc0b7080 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b7de0 con 0x7fcebc0b9760 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcec1e4e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b8b20 con 0x7fcebc15dbc0 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54668/0 (socket says 172.21.15.12:54668) 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.092+0000 7fcebb7fe640 1 -- 172.21.15.12:0/1675490348 learned_addr learned my addr 172.21.15.12:0/1675490348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebbfff640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 0x7fcebc15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 0x7fcebc160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 -- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 msgr2=0x7fcebc15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 0x7fcebc15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 -- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 msgr2=0x7fcebc160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 0x7fcebc160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 -- 172.21.15.12:0/1675490348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc1e8210 con 0x7fcebc0b7080 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 0x7fcebc160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebbfff640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 0x7fcebc15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcebb7fe640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fceac00c430 tx=0x7fceac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac0074e0 con 0x7fcebc0b7080 2024-03-08T08:31:43.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac0047e0 con 0x7fcebc0b7080 2024-03-08T08:31:43.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc1e84f0 con 0x7fcebc0b7080 2024-03-08T08:31:43.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac004bd0 con 0x7fcebc0b7080 2024-03-08T08:31:43.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.093+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1e8ab0 con 0x7fcebc0b7080 2024-03-08T08:31:43.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.094+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0cc600 con 0x7fcebc0b7080 2024-03-08T08:31:43.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.094+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceac011070 con 0x7fcebc0b7080 2024-03-08T08:31:43.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.095+0000 7fcea3fff640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 0x7fcea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.095+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fceac089b70 con 0x7fcebc0b7080 2024-03-08T08:31:43.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.095+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 0x7fcea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.095+0000 7fcebaffd640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 0x7fcea806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fceb0036000 tx=0x7fceb0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.099+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fceac058cf0 con 0x7fcebc0b7080 2024-03-08T08:31:43.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.279+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcebc0749c0 con 0x7fcea806b4d0 2024-03-08T08:31:43.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.279+0000 7fcea3fff640 1 -- 172.21.15.12:0/1675490348 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcebc0749c0 con 0x7fcea806b4d0 2024-03-08T08:31:43.281 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 3 to deep-scrub 2024-03-08T08:31:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 msgr2=0x7fcea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 0x7fcea806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fceb0036000 tx=0x7fceb0008030 comp rx=0 tx=0).stop 2024-03-08T08:31:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 msgr2=0x7fcebc0caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fceac00c430 tx=0x7fceac00c8f0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 shutdown_connections 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fcea806b4d0 0x7fcea806d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcebc15dbc0 0x7fcebc15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcebc0b9760 0x7fcebc160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 --2- 172.21.15.12:0/1675490348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcebc0b7080 0x7fcebc0caba0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.281+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 >> 172.21.15.12:0/1675490348 conn(0x7fcebc0726f0 msgr2=0x7fcebc15d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.282+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 shutdown_connections 2024-03-08T08:31:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.282+0000 7fcec1e4e640 1 -- 172.21.15.12:0/1675490348 wait complete. 2024-03-08T08:31:43.293 INFO:tasks.ceph:Scrubbing osd.4 2024-03-08T08:31:43.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.398+0000 7fbb47593640 1 Processor -- start 2024-03-08T08:31:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.398+0000 7fbb47593640 1 -- start start 2024-03-08T08:31:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb47593640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbb400cc410 0x7fbb400c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb47593640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb400cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb47593640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb400cd680 con 0x7fbb4015f330 2024-03-08T08:31:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb47593640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb400d1780 con 0x7fbb400cc410 2024-03-08T08:31:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb47593640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400cc8a0 con 0x7fbb400c5990 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb4532f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbb4015f330 0x7fbb4009a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37048/0 (socket says 172.21.15.12:37048) 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.399+0000 7fbb4532f640 1 -- 172.21.15.12:0/2300856363 learned_addr learned my addr 172.21.15.12:0/2300856363 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965359875 0 0) 0x7fbb400cd680 con 0x7fbb4015f330 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb28003520 con 0x7fbb4015f330 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2300856363 >> v2:172.21.15.156:3300/0 conn(0x7fbb400cc410 0x7fbb400c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb45b30640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb400cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3324074214 0 0) 0x7fbb28003520 con 0x7fbb4015f330 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 msgr2=0x7fbb400cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb400cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 >> v2:172.21.15.156:3300/0 conn(0x7fbb400cc410 msgr2=0x7fbb400c5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 --2- 172.21.15.12:0/2300856363 >> v2:172.21.15.156:3300/0 conn(0x7fbb400cc410 0x7fbb400c5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40099ef0 con 0x7fbb4015f330 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7fbb300019f0 con 0x7fbb4015f330 2024-03-08T08:31:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 >> v1:172.21.15.12:6789/0 conn(0x7fbb4015f330 legacy=0x7fbb4009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb28003f60 0x7fbb28004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28004d40 0x7fbb280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400d1780 con 0x7fbb28003f60 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400cc8a0 con 0x7fbb28004d40 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2800ba40 con 0x7fbb400c5990 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb28003f60 0x7fbb28004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb4532f640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb45b30640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28004d40 0x7fbb280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 -- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 msgr2=0x7fbb280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb400c5990 0x7fbb280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 -- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28004d40 msgr2=0x7fbb280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28004d40 0x7fbb280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.400+0000 7fbb44b2e640 1 -- 172.21.15.12:0/2300856363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28003520 con 0x7fbb28003f60 2024-03-08T08:31:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.401+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb28003f60 0x7fbb28004800 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009ac0 tx=0x7fbb28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf88d2ca016db1c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.401+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34039070 con 0x7fbb28003f60 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.401+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3402c710 con 0x7fbb28003f60 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.401+0000 7fbb3ffff640 1 -- 172.21.15.12:0/2300856363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34009690 con 0x7fbb28003f60 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 -- 172.21.15.12:0/2300856363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb28003f60 msgr2=0x7fbb28004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 --2- 172.21.15.12:0/2300856363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb28003f60 0x7fbb28004800 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009ac0 tx=0x7fbb28005830 comp rx=0 tx=0).stop 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb45b30640 1 -- 172.21.15.12:0/2300856363 reap_dead start 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 -- 172.21.15.12:0/2300856363 shutdown_connections 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 -- 172.21.15.12:0/2300856363 >> 172.21.15.12:0/2300856363 conn(0x7fbb400d1900 msgr2=0x7fbb400d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 -- 172.21.15.12:0/2300856363 shutdown_connections 2024-03-08T08:31:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.402+0000 7fbb47593640 1 -- 172.21.15.12:0/2300856363 wait complete. 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.403+0000 7fbb47593640 1 Processor -- start 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.403+0000 7fbb47593640 1 -- start start 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28003f60 0x7fbb40153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb4015c6b0 0x7fbb400ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 0x7fbb4015d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400cd430 con 0x7fbb400cae10 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40099ca0 con 0x7fbb28003f60 2024-03-08T08:31:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4009a540 con 0x7fbb4015c6b0 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 0x7fbb4015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 0x7fbb4015d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54690/0 (socket says 172.21.15.12:54690) 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 -- 172.21.15.12:0/2528552011 learned_addr learned my addr 172.21.15.12:0/2528552011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb4532f640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28003f60 0x7fbb40153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 -- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb4015c6b0 msgr2=0x7fbb400ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb4015c6b0 0x7fbb400ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb4015c6b0 0x7fbb400ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 -- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28003f60 msgr2=0x7fbb40153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28003f60 0x7fbb40153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4015b8f0 con 0x7fbb400cae10 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb4532f640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb28003f60 0x7fbb40153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb4015c6b0 0x7fbb400ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb45b30640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 0x7fbb4015d330 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbb400b33e0 tx=0x7fbb38011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38019020 con 0x7fbb400cae10 2024-03-08T08:31:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb38004030 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38004400 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.404+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4015bbd0 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.405+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4015c190 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.405+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb400749c0 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.406+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb3801c020 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.406+0000 7fbb3dffb640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb2c06b500 0x7fbb2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.406+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbb3808a270 con 0x7fbb400cae10 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb4532f640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb2c06b500 0x7fbb2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb3dffb640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7fbb2c0713c0 0x7fbb2c0737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb2c073e90 con 0x7fbb2c0713c0 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb38091050 con 0x7fbb400cae10 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb4532f640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb2c06b500 0x7fbb2c06d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbb30004b60 tx=0x7fbb30006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7fbb2c0713c0 0x7fbb2c0737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.407+0000 7fbb44b2e640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7fbb2c0713c0 0x7fbb2c0737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.408+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== osd.4 v2:172.21.15.156:6816/602210282 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbb2c073e90 con 0x7fbb2c0713c0 2024-03-08T08:31:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.421+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 --> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbb400ce0f0 con 0x7fbb2c0713c0 2024-03-08T08:31:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb3dffb640 1 -- 172.21.15.12:0/2528552011 <== osd.4 v2:172.21.15.156:6816/602210282 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbb400ce0f0 con 0x7fbb2c0713c0 2024-03-08T08:31:43.423 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:43.423 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7fbb2c0713c0 msgr2=0x7fbb2c0737e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.156:6816/602210282,v1:172.21.15.156:6817/602210282] conn(0x7fbb2c0713c0 0x7fbb2c0737e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb2c06b500 msgr2=0x7fbb2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fbb2c06b500 0x7fbb2c06d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbb30004b60 tx=0x7fbb30006040 comp rx=0 tx=0).stop 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 msgr2=0x7fbb4015d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 --2- 172.21.15.12:0/2528552011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb400cae10 0x7fbb4015d330 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbb400b33e0 tx=0x7fbb38011cf0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb45b30640 1 -- 172.21.15.12:0/2528552011 reap_dead start 2024-03-08T08:31:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 shutdown_connections 2024-03-08T08:31:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 >> 172.21.15.12:0/2528552011 conn(0x7fbb400d1900 msgr2=0x7fbb400c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 shutdown_connections 2024-03-08T08:31:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.422+0000 7fbb47593640 1 -- 172.21.15.12:0/2528552011 wait complete. 2024-03-08T08:31:43.434 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-08T08:31:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.534+0000 7fefc0581640 1 Processor -- start 2024-03-08T08:31:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.534+0000 7fefc0581640 1 -- start start 2024-03-08T08:31:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.535+0000 7fefc0581640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.535+0000 7fefc0581640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 0x7fefb806b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.535+0000 7fefc0581640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb815c850 con 0x7fefb80cff30 2024-03-08T08:31:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.535+0000 7fefc0581640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb815c620 con 0x7fefb806abd0 2024-03-08T08:31:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.535+0000 7fefc0581640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb806b560 con 0x7fefb80cfb50 2024-03-08T08:31:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbeb1e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 0x7fefb806b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbeb1e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 0x7fefb806b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.12:49786/0 (socket says 172.21.15.12:49786) 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbdb1c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fefb80cff30 0x7fefb815fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.12:36548/0 (socket says 172.21.15.12:36548) 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.12:38608/0 (socket says 172.21.15.12:38608) 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbeb1e640 1 -- 172.21.15.12:0/2092968176 learned_addr learned my addr 172.21.15.12:0/2092968176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 -- 172.21.15.12:0/2092968176 >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 msgr2=0x7fefb806b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 --2- 172.21.15.12:0/2092968176 >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 0x7fefb806b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 -- 172.21.15.12:0/2092968176 >> v1:172.21.15.156:6789/0 conn(0x7fefb80cff30 legacy=0x7fefb815fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.536+0000 7fefbe31d640 1 -- 172.21.15.12:0/2092968176 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb815cb30 con 0x7fefb80cfb50 2024-03-08T08:31:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefbe31d640 1 --2- 172.21.15.12:0/2092968176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fefb00097a0 tx=0x7fefb002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abdb07d042526f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefbd31b640 1 -- 172.21.15.12:0/2092968176 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0035020 con 0x7fefb80cfb50 2024-03-08T08:31:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefbd31b640 1 -- 172.21.15.12:0/2092968176 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb0031830 con 0x7fefb80cfb50 2024-03-08T08:31:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefbd31b640 1 -- 172.21.15.12:0/2092968176 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0031c00 con 0x7fefb80cfb50 2024-03-08T08:31:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefc0581640 1 -- 172.21.15.12:0/2092968176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 msgr2=0x7fefb815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.537+0000 7fefc0581640 1 --2- 172.21.15.12:0/2092968176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fefb00097a0 tx=0x7fefb002b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 -- 172.21.15.12:0/2092968176 shutdown_connections 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 --2- 172.21.15.12:0/2092968176 >> v2:172.21.15.156:3300/0 conn(0x7fefb806abd0 0x7fefb806b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 --2- 172.21.15.12:0/2092968176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb815c0e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 -- 172.21.15.12:0/2092968176 >> 172.21.15.12:0/2092968176 conn(0x7fefb8099a40 msgr2=0x7fefb80b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 -- 172.21.15.12:0/2092968176 shutdown_connections 2024-03-08T08:31:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.538+0000 7fefc0581640 1 -- 172.21.15.12:0/2092968176 wait complete. 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 Processor -- start 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 -- start start 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 0x7fefb81f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb80cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80d0390 con 0x7fefb80cff30 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb815f870 con 0x7fefb806abd0 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.539+0000 7fefc0581640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c5b30 con 0x7fefb80cfb50 2024-03-08T08:31:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54694/0 (socket says 172.21.15.12:54694) 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 -- 172.21.15.12:0/938018880 learned_addr learned my addr 172.21.15.12:0/938018880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbe31d640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 0x7fefb81f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbdb1c640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb80cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 -- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 msgr2=0x7fefb80cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb80cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 -- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 msgr2=0x7fefb81f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 0x7fefb81f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 -- 172.21.15.12:0/938018880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80c5e10 con 0x7fefb80cff30 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbe31d640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 0x7fefb81f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbdb1c640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb80cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefbeb1e640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fefa800efc0 tx=0x7fefa800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa800cd20 con 0x7fefb80cff30 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa8003990 con 0x7fefb80cff30 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8003d80 con 0x7fefb80cff30 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb80c60f0 con 0x7fefb80cff30 2024-03-08T08:31:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.540+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb81f5b40 con 0x7fefb80cff30 2024-03-08T08:31:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.541+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80749c0 con 0x7fefb80cff30 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.542+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefa8012050 con 0x7fefb80cff30 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.542+0000 7fefb6ffd640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 0x7fefa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.542+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fefa808a3f0 con 0x7fefb80cff30 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.542+0000 7fefbe31d640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 0x7fefa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.543+0000 7fefbe31d640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 0x7fefa406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fefb0036000 tx=0x7fefb003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.547+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fefa8010040 con 0x7fefb80cff30 2024-03-08T08:31:43.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.728+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fefb80aff80 con 0x7fefa406b4d0 2024-03-08T08:31:43.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.728+0000 7fefb6ffd640 1 -- 172.21.15.12:0/938018880 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fefb80aff80 con 0x7fefa406b4d0 2024-03-08T08:31:43.729 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 4 to deep-scrub 2024-03-08T08:31:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 msgr2=0x7fefa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 0x7fefa406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fefb0036000 tx=0x7fefb003c040 comp rx=0 tx=0).stop 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 msgr2=0x7fefb81567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fefa800efc0 tx=0x7fefa800c490 comp rx=0 tx=0).stop 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 shutdown_connections 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fefa406b4d0 0x7fefa406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb80cff30 0x7fefb81567a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefb80cfb50 0x7fefb80cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 --2- 172.21.15.12:0/938018880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefb806abd0 0x7fefb81f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 >> 172.21.15.12:0/938018880 conn(0x7fefb8099a40 msgr2=0x7fefb815e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 shutdown_connections 2024-03-08T08:31:43.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.730+0000 7fefc0581640 1 -- 172.21.15.12:0/938018880 wait complete. 2024-03-08T08:31:43.742 INFO:tasks.ceph:Scrubbing osd.5 2024-03-08T08:31:43.742 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.846+0000 7f08819a2640 1 Processor -- start 2024-03-08T08:31:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.846+0000 7f08819a2640 1 -- start start 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08819a2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f087c0b0d80 0x7f087c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08819a2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0cf9c0 con 0x7f087c0b09a0 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08819a2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0cfca0 con 0x7f087c09a1c0 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08819a2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0cff80 con 0x7f087c0b0d80 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08809a0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f087c0b09a0 0x7f087c099ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37052/0 (socket says 172.21.15.12:37052) 2024-03-08T08:31:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f08809a0640 1 -- 172.21.15.12:0/1525021862 learned_addr learned my addr 172.21.15.12:0/1525021862 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708506360 0 0) 0x7f087c0cf9c0 con 0x7f087c0b09a0 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c003520 con 0x7f087c0b09a0 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087bfff640 1 --2- 172.21.15.12:0/1525021862 >> v2:172.21.15.156:3300/0 conn(0x7f087c0b0d80 0x7f087c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2731866888 0 0) 0x7f086c003520 con 0x7f087c0b09a0 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 >> v2:172.21.15.156:3300/0 conn(0x7f087c0b0d80 msgr2=0x7f087c0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 --2- 172.21.15.12:0/1525021862 >> v2:172.21.15.156:3300/0 conn(0x7f087c0b0d80 0x7f087c0cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 >> v1:172.21.15.156:6789/0 conn(0x7f087c09a1c0 legacy=0x7f087c15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0d0260 con 0x7f087c0b09a0 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f0870001670 con 0x7f087c0b09a0 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 >> v1:172.21.15.12:6789/0 conn(0x7f087c0b09a0 legacy=0x7f087c099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.847+0000 7f087b7fe640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0039b0 0x7f086c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087b7fe640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 0x7f086c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087b7fe640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c0051c0 0x7f086c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cff80 con 0x7f086c004800 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c00c160 con 0x7f086c0039b0 2024-03-08T08:31:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c00c360 con 0x7f086c0051c0 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 0x7f086c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f08809a0640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0039b0 0x7f086c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f08811a1640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c0051c0 0x7f086c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 -- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c0051c0 msgr2=0x7f086c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c0051c0 0x7f086c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 -- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0039b0 msgr2=0x7f086c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0039b0 0x7f086c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f087bfff640 1 -- 172.21.15.12:0/1525021862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c003520 con 0x7f086c004800 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.848+0000 7f08811a1640 1 -- 172.21.15.12:0/1525021862 reap_dead start 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f087bfff640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 0x7f086c004c80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0868009a90 tx=0x7f086c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a5b6c6f6706f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868039070 con 0x7f086c004800 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086802c6b0 con 0x7f086c004800 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f087b7fe640 1 -- 172.21.15.12:0/1525021862 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868009690 con 0x7f086c004800 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 -- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 msgr2=0x7f086c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 0x7f086c004c80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0868009a90 tx=0x7f086c005830 comp rx=0 tx=0).stop 2024-03-08T08:31:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 -- 172.21.15.12:0/1525021862 shutdown_connections 2024-03-08T08:31:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 --2- 172.21.15.12:0/1525021862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f086c004800 0x7f086c004c80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 -- 172.21.15.12:0/1525021862 >> 172.21.15.12:0/1525021862 conn(0x7f087c0d16f0 msgr2=0x7f087c0d1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 -- 172.21.15.12:0/1525021862 shutdown_connections 2024-03-08T08:31:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.849+0000 7f08819a2640 1 -- 172.21.15.12:0/1525021862 wait complete. 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 Processor -- start 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 -- start start 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c004800 0x7f087c0b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 0x7f087c0c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f087c0c9130 0x7f087c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cd550 con 0x7f087c155e80 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cd3d0 con 0x7f087c0c9130 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.851+0000 7f08819a2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cd6d0 con 0x7f086c004800 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 0x7f087c0c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 0x7f087c0c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54712/0 (socket says 172.21.15.12:54712) 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 -- 172.21.15.12:0/1792990534 learned_addr learned my addr 172.21.15.12:0/1792990534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08809a0640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c004800 0x7f087c0b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08811a1640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f087c0c9130 0x7f087c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 -- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c004800 msgr2=0x7f087c0b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f086c004800 0x7f087c0b58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 -- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f087c0c9130 msgr2=0x7f087c0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f087c0c9130 0x7f087c0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c155c60 con 0x7f087c155e80 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08811a1640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f087c0c9130 0x7f087c0c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f087bfff640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 0x7f087c0c8bf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f086802c320 tx=0x7f0868004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868042030 con 0x7f087c155e80 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086803a040 con 0x7f087c155e80 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c0b6a20 con 0x7f087c155e80 2024-03-08T08:31:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868039070 con 0x7f087c155e80 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.852+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c0b6fe0 con 0x7f087c155e80 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.853+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f087c156640 con 0x7f087c155e80 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0868045070 con 0x7f087c155e80 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086006b500 0x7f086006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f08680aabf0 con 0x7f087c155e80 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08809a0640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086006b500 0x7f086006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f08600713c0 0x7f08600737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0868036e90 con 0x7f08600713c0 2024-03-08T08:31:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08680b1020 con 0x7f087c155e80 2024-03-08T08:31:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08811a1640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f08600713c0 0x7f08600737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.854+0000 7f08809a0640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086006b500 0x7f086006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08700011e0 tx=0x7f0870001150 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.855+0000 7f08811a1640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f08600713c0 0x7f08600737e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.856+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== osd.5 v2:172.21.15.156:6824/1688693160 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0868036e90 con 0x7f08600713c0 2024-03-08T08:31:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.869+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 --> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f087c0749c0 con 0x7f08600713c0 2024-03-08T08:31:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.869+0000 7f08797fa640 1 -- 172.21.15.12:0/1792990534 <== osd.5 v2:172.21.15.156:6824/1688693160 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f087c0749c0 con 0x7f08600713c0 2024-03-08T08:31:43.871 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:43.871 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f08600713c0 msgr2=0x7f08600737e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.156:6824/1688693160,v1:172.21.15.156:6825/1688693160] conn(0x7f08600713c0 0x7f08600737e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086006b500 msgr2=0x7f086006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f086006b500 0x7f086006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08700011e0 tx=0x7f0870001150 comp rx=0 tx=0).stop 2024-03-08T08:31:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 msgr2=0x7f087c0c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 --2- 172.21.15.12:0/1792990534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f087c155e80 0x7f087c0c8bf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f086802c320 tx=0x7f0868004730 comp rx=0 tx=0).stop 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08811a1640 1 -- 172.21.15.12:0/1792990534 reap_dead start 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 shutdown_connections 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 >> 172.21.15.12:0/1792990534 conn(0x7f087c0d16f0 msgr2=0x7f087c0af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 shutdown_connections 2024-03-08T08:31:43.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.870+0000 7f08819a2640 1 -- 172.21.15.12:0/1792990534 wait complete. 2024-03-08T08:31:43.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-08T08:31:43.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.982+0000 7f7df11b5640 1 Processor -- start 2024-03-08T08:31:43.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.982+0000 7f7df11b5640 1 -- start start 2024-03-08T08:31:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7dec15ddc0 0x7f7dec0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7dec0d26f0 0x7f7dec0d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0b9030 con 0x7f7dec0d26f0 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0d13a0 con 0x7f7dec15ddc0 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7df11b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0d1d20 con 0x7f7dec0d0f40 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7dec0d26f0 0x7f7dec0d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7dec0d26f0 0x7f7dec0d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54720/0 (socket says 172.21.15.12:54720) 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 -- 172.21.15.12:0/1347493614 learned_addr learned my addr 172.21.15.12:0/1347493614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7dead9d640 1 --2- 172.21.15.12:0/1347493614 >> v2:172.21.15.156:3300/0 conn(0x7f7dec15ddc0 0x7f7dec0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7dea59c640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 -- 172.21.15.12:0/1347493614 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 msgr2=0x7f7dec0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 -- 172.21.15.12:0/1347493614 >> v2:172.21.15.156:3300/0 conn(0x7f7dec15ddc0 msgr2=0x7f7dec0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 --2- 172.21.15.12:0/1347493614 >> v2:172.21.15.156:3300/0 conn(0x7f7dec15ddc0 0x7f7dec0d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.983+0000 7f7deb59e640 1 -- 172.21.15.12:0/1347493614 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec15e1a0 con 0x7f7dec0d26f0 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.984+0000 7f7deb59e640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec0d26f0 0x7f7dec0d2b80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0b66a0 tx=0x7f7ddc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cb6bb8b2f0d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.984+0000 7f7de9d9b640 1 -- 172.21.15.12:0/1347493614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc02ca70 con 0x7f7dec0d26f0 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.984+0000 7f7de9d9b640 1 -- 172.21.15.12:0/1347493614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc02cc10 con 0x7f7dec0d26f0 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.984+0000 7f7de9d9b640 1 -- 172.21.15.12:0/1347493614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0343f0 con 0x7f7dec0d26f0 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 -- 172.21.15.12:0/1347493614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec0d26f0 msgr2=0x7f7dec0d2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec0d26f0 0x7f7dec0d2b80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0b66a0 tx=0x7f7ddc02bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 -- 172.21.15.12:0/1347493614 shutdown_connections 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec0d26f0 0x7f7dec0d2b80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 --2- 172.21.15.12:0/1347493614 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 --2- 172.21.15.12:0/1347493614 >> v2:172.21.15.156:3300/0 conn(0x7f7dec15ddc0 0x7f7dec0d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 -- 172.21.15.12:0/1347493614 >> 172.21.15.12:0/1347493614 conn(0x7f7dec072720 msgr2=0x7f7dec072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 -- 172.21.15.12:0/1347493614 shutdown_connections 2024-03-08T08:31:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.985+0000 7f7df11b5640 1 -- 172.21.15.12:0/1347493614 wait complete. 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.986+0000 7f7df11b5640 1 Processor -- start 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.986+0000 7f7df11b5640 1 -- start start 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dec0d26f0 0x7f7dec14a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b8d50 con 0x7f7dec15ddc0 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b8a70 con 0x7f7dec0d26f0 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7df11b5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b9980 con 0x7f7dec0d0f40 2024-03-08T08:31:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54736/0 (socket says 172.21.15.12:54736) 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 -- 172.21.15.12:0/3353879485 learned_addr learned my addr 172.21.15.12:0/3353879485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7dead9d640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7dea59c640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dec0d26f0 0x7f7dec14a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 -- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 msgr2=0x7f7dec0c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 -- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dec0d26f0 msgr2=0x7f7dec14a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dec0d26f0 0x7f7dec14a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 -- 172.21.15.12:0/3353879485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec14ae20 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.987+0000 7f7deb59e640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009f40 tx=0x7f7ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.988+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc043020 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.988+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc03e450 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.988+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec15f8d0 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.988+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc009510 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.988+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec15fe90 con 0x7f7dec15ddc0 2024-03-08T08:31:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.989+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec0749c0 con 0x7f7dec15ddc0 2024-03-08T08:31:43.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.989+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ddc046020 con 0x7f7dec15ddc0 2024-03-08T08:31:43.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.990+0000 7f7dd37fe640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 0x7f7dd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:43.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.990+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ddc0ac7d0 con 0x7f7dec15ddc0 2024-03-08T08:31:43.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.990+0000 7f7dead9d640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 0x7f7dd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:43.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.990+0000 7f7dead9d640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 0x7f7dd806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7de4006fd0 tx=0x7f7de4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:43.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:43.994+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7ddc04d030 con 0x7f7dec15ddc0 2024-03-08T08:31:44.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.177+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dec0cf9b0 con 0x7f7dd806b4d0 2024-03-08T08:31:44.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.178+0000 7f7dd37fe640 1 -- 172.21.15.12:0/3353879485 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7dec0cf9b0 con 0x7f7dd806b4d0 2024-03-08T08:31:44.180 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 5 to deep-scrub 2024-03-08T08:31:44.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 msgr2=0x7f7dd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 0x7f7dd806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7de4006fd0 tx=0x7f7de4008040 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 msgr2=0x7f7dec0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009f40 tx=0x7f7ddc004a40 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 shutdown_connections 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f7dd806b4d0 0x7f7dd806d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dec15ddc0 0x7f7dec0b4650 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dec0d26f0 0x7f7dec14a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 --2- 172.21.15.12:0/3353879485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7dec0d0f40 0x7f7dec0c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 >> 172.21.15.12:0/3353879485 conn(0x7f7dec072720 msgr2=0x7f7dec0ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 shutdown_connections 2024-03-08T08:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.180+0000 7f7df11b5640 1 -- 172.21.15.12:0/3353879485 wait complete. 2024-03-08T08:31:44.192 INFO:tasks.ceph:Scrubbing osd.6 2024-03-08T08:31:44.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.297+0000 7f7115313640 1 Processor -- start 2024-03-08T08:31:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.297+0000 7f7115313640 1 -- start start 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f7115313640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f711009a310 0x7f711006f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f7115313640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f711006bce0 0x7f711006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f7115313640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711006c230 con 0x7f711015db70 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f7115313640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711006ce40 con 0x7f711009a310 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f7115313640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711006c510 con 0x7f711006bce0 2024-03-08T08:31:44.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f711009a310 0x7f711006f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f711015db70 0x7f7110099c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37056/0 (socket says 172.21.15.12:37056) 2024-03-08T08:31:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710effd640 1 -- 172.21.15.12:0/1372720056 learned_addr learned my addr 172.21.15.12:0/1372720056 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1372720056 >> v2:172.21.15.12:3300/0 conn(0x7f711009a310 0x7f711006f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54750/0 (socket says 172.21.15.12:54750) 2024-03-08T08:31:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710f7fe640 1 --2- 172.21.15.12:0/1372720056 >> v2:172.21.15.156:3300/0 conn(0x7f711006bce0 0x7f711006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 -- 172.21.15.12:0/1372720056 >> v2:172.21.15.156:3300/0 conn(0x7f711006bce0 msgr2=0x7f711006fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1372720056 >> v2:172.21.15.156:3300/0 conn(0x7f711006bce0 0x7f711006fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 -- 172.21.15.12:0/1372720056 >> v1:172.21.15.12:6789/0 conn(0x7f711015db70 legacy=0x7f7110099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710e7fc640 1 -- 172.21.15.12:0/1372720056 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711006c7f0 con 0x7f711009a310 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.298+0000 7f710dffb640 1 -- 172.21.15.12:0/1372720056 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229216888 0 0) 0x7f711006c230 con 0x7f711015db70 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1372720056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711009a310 0x7f711006f750 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f71100b1350 tx=0x7f710402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1bc93d1605c790 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f710dffb640 1 -- 172.21.15.12:0/1372720056 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104035020 con 0x7f711009a310 2024-03-08T08:31:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f710dffb640 1 -- 172.21.15.12:0/1372720056 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7104031830 con 0x7f711009a310 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f710dffb640 1 -- 172.21.15.12:0/1372720056 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104031c00 con 0x7f711009a310 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f7115313640 1 -- 172.21.15.12:0/1372720056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711009a310 msgr2=0x7f711006f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f7115313640 1 --2- 172.21.15.12:0/1372720056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711009a310 0x7f711006f750 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f71100b1350 tx=0x7f710402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.299+0000 7f7115313640 1 -- 172.21.15.12:0/1372720056 shutdown_connections 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.300+0000 7f7115313640 1 --2- 172.21.15.12:0/1372720056 >> v2:172.21.15.156:3300/0 conn(0x7f711006bce0 0x7f711006fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.300+0000 7f7115313640 1 --2- 172.21.15.12:0/1372720056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711009a310 0x7f711006f750 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.300+0000 7f7115313640 1 -- 172.21.15.12:0/1372720056 >> 172.21.15.12:0/1372720056 conn(0x7f71100b3430 msgr2=0x7f71100b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.300+0000 7f7115313640 1 -- 172.21.15.12:0/1372720056 shutdown_connections 2024-03-08T08:31:44.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.300+0000 7f7115313640 1 -- 172.21.15.12:0/1372720056 wait complete. 2024-03-08T08:31:44.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.301+0000 7f7115313640 1 Processor -- start 2024-03-08T08:31:44.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.301+0000 7f7115313640 1 -- start start 2024-03-08T08:31:44.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f711006bce0 0x7f71100cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f711009a310 0x7f71100cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 0x7f71101f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015e0d0 con 0x7f711015db70 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015df50 con 0x7f711006bce0 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100701d0 con 0x7f711009a310 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 0x7f71101f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 0x7f71101f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54764/0 (socket says 172.21.15.12:54764) 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 -- 172.21.15.12:0/1425568639 learned_addr learned my addr 172.21.15.12:0/1425568639 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710effd640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f711006bce0 0x7f71100cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f711009a310 0x7f71100cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 -- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f711009a310 msgr2=0x7f71100cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f711009a310 0x7f71100cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 -- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f711006bce0 msgr2=0x7f71100cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f711006bce0 0x7f71100cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711015e750 con 0x7f711015db70 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f710f7fe640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 0x7f71101f6f20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7108007e40 tx=0x7f710800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108014620 con 0x7f711015db70 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7108019070 con 0x7f711015db70 2024-03-08T08:31:44.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7110145cb0 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108014a20 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.302+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7110146270 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.303+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71100749c0 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7108014bc0 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70fc06b4d0 0x7f70fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f710808a870 con 0x7f711015db70 2024-03-08T08:31:44.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f710effd640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70fc06b4d0 0x7f70fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7f70fc071390 0x7f70fc0737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70fc073e60 con 0x7f70fc071390 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f710808f2c0 con 0x7f711015db70 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7f70fc071390 0x7f70fc0737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.304+0000 7f710effd640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70fc06b4d0 0x7f70fc06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7100000f60 tx=0x7f7100007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.305+0000 7f710e7fc640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7f70fc071390 0x7f70fc0737b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.306+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== osd.6 v2:172.21.15.156:6800/2822268422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f70fc073e60 con 0x7f70fc071390 2024-03-08T08:31:44.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.319+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 --> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f711015e4a0 con 0x7f70fc071390 2024-03-08T08:31:44.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f70f77fe640 1 -- 172.21.15.12:0/1425568639 <== osd.6 v2:172.21.15.156:6800/2822268422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f711015e4a0 con 0x7f70fc071390 2024-03-08T08:31:44.321 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:44.321 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:44.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7f70fc071390 msgr2=0x7f70fc0737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.156:6800/2822268422,v1:172.21.15.156:6801/2822268422] conn(0x7f70fc071390 0x7f70fc0737b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70fc06b4d0 msgr2=0x7f70fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f70fc06b4d0 0x7f70fc06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7100000f60 tx=0x7f7100007c00 comp rx=0 tx=0).stop 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 msgr2=0x7f71101f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 --2- 172.21.15.12:0/1425568639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711015db70 0x7f71101f6f20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7108007e40 tx=0x7f710800e4b0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f710f7fe640 1 -- 172.21.15.12:0/1425568639 reap_dead start 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 shutdown_connections 2024-03-08T08:31:44.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 >> 172.21.15.12:0/1425568639 conn(0x7f71100b3430 msgr2=0x7f71100ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 shutdown_connections 2024-03-08T08:31:44.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.320+0000 7f7115313640 1 -- 172.21.15.12:0/1425568639 wait complete. 2024-03-08T08:31:44.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-08T08:31:44.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 Processor -- start 2024-03-08T08:31:44.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 -- start start 2024-03-08T08:31:44.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f475015ddc0 0x7f47500d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 0x7f47500d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47500d26f0 0x7f47500d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47500b9030 con 0x7f47500d26f0 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47500d13a0 con 0x7f47500d0f40 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f475702e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47500d1d20 con 0x7f475015ddc0 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.433+0000 7f47555cb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47500d26f0 0x7f47500d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47500d26f0 0x7f47500d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54780/0 (socket says 172.21.15.12:54780) 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 -- 172.21.15.12:0/407044649 learned_addr learned my addr 172.21.15.12:0/407044649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f474ffff640 1 --2- 172.21.15.12:0/407044649 >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 0x7f47500d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 -- 172.21.15.12:0/407044649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f475015ddc0 msgr2=0x7f47500d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 --2- 172.21.15.12:0/407044649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f475015ddc0 0x7f47500d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 -- 172.21.15.12:0/407044649 >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 msgr2=0x7f47500d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 --2- 172.21.15.12:0/407044649 >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 0x7f47500d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f47555cb640 1 -- 172.21.15.12:0/407044649 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475015e1a0 con 0x7f47500d26f0 2024-03-08T08:31:44.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.434+0000 7f474ffff640 1 --2- 172.21.15.12:0/407044649 >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 0x7f47500d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:44.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f47555cb640 1 --2- 172.21.15.12:0/407044649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d26f0 0x7f47500d2b80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f475008ccc0 tx=0x7f474802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac5e8ff4373ebb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f474f7fe640 1 -- 172.21.15.12:0/407044649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474802ca70 con 0x7f47500d26f0 2024-03-08T08:31:44.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f474f7fe640 1 -- 172.21.15.12:0/407044649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474802cc10 con 0x7f47500d26f0 2024-03-08T08:31:44.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f474f7fe640 1 -- 172.21.15.12:0/407044649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480343f0 con 0x7f47500d26f0 2024-03-08T08:31:44.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 -- 172.21.15.12:0/407044649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d26f0 msgr2=0x7f47500d2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 --2- 172.21.15.12:0/407044649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d26f0 0x7f47500d2b80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f475008ccc0 tx=0x7f474802bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 -- 172.21.15.12:0/407044649 shutdown_connections 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 --2- 172.21.15.12:0/407044649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d26f0 0x7f47500d2b80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 --2- 172.21.15.12:0/407044649 >> v2:172.21.15.156:3300/0 conn(0x7f47500d0f40 0x7f47500d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 --2- 172.21.15.12:0/407044649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f475015ddc0 0x7f47500d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 -- 172.21.15.12:0/407044649 >> 172.21.15.12:0/407044649 conn(0x7f4750072720 msgr2=0x7f4750072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 -- 172.21.15.12:0/407044649 shutdown_connections 2024-03-08T08:31:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.435+0000 7f475702e640 1 -- 172.21.15.12:0/407044649 wait complete. 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.436+0000 7f475702e640 1 Processor -- start 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.436+0000 7f475702e640 1 -- start start 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 0x7f475015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f475015ddc0 0x7f475014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b8d50 con 0x7f47500d0f40 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b8a70 con 0x7f475015ddc0 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f475702e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b9980 con 0x7f47500d26f0 2024-03-08T08:31:44.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f4754dca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f4754dca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54792/0 (socket says 172.21.15.12:54792) 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.437+0000 7f4754dca640 1 -- 172.21.15.12:0/2274490425 learned_addr learned my addr 172.21.15.12:0/2274490425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f47555cb640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f475015ddc0 0x7f475014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f474ffff640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 0x7f475015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 -- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 msgr2=0x7f475015f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 0x7f475015f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 -- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f475015ddc0 msgr2=0x7f475014ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f475015ddc0 0x7f475014ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 -- 172.21.15.12:0/2274490425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475015f900 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f474ffff640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 0x7f475015f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f4754dca640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f474400b410 tx=0x7f474400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744015020 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744011850 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475015fb80 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744011c40 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.438+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4750160140 con 0x7f47500d0f40 2024-03-08T08:31:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.439+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47500749c0 con 0x7f47500d0f40 2024-03-08T08:31:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.440+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4744018020 con 0x7f47500d0f40 2024-03-08T08:31:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.440+0000 7f474d7fa640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.440+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4744089e30 con 0x7f47500d0f40 2024-03-08T08:31:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.440+0000 7f474ffff640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.440+0000 7f474ffff640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 0x7f473406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f473c006fd0 tx=0x7f473c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.444+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f474401f020 con 0x7f47500d0f40 2024-03-08T08:31:44.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.624+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f4750160310 con 0x7f473406b4d0 2024-03-08T08:31:44.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.625+0000 7f474d7fa640 1 -- 172.21.15.12:0/2274490425 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4750160310 con 0x7f473406b4d0 2024-03-08T08:31:44.626 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 6 to deep-scrub 2024-03-08T08:31:44.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 msgr2=0x7f473406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 0x7f473406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f473c006fd0 tx=0x7f473c008040 comp rx=0 tx=0).stop 2024-03-08T08:31:44.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 msgr2=0x7f4750160ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f474400b410 tx=0x7f474400b8d0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 shutdown_connections 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f475015ddc0 0x7f475014ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47500d26f0 0x7f475015f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 --2- 172.21.15.12:0/2274490425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47500d0f40 0x7f4750160ed0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 >> 172.21.15.12:0/2274490425 conn(0x7f4750072720 msgr2=0x7f47500ce510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 shutdown_connections 2024-03-08T08:31:44.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.627+0000 7f475702e640 1 -- 172.21.15.12:0/2274490425 wait complete. 2024-03-08T08:31:44.639 INFO:tasks.ceph:Scrubbing osd.7 2024-03-08T08:31:44.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-08T08:31:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.743+0000 7fa40373e640 1 Processor -- start 2024-03-08T08:31:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.743+0000 7fa40373e640 1 -- start start 2024-03-08T08:31:44.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa40373e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa40373e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 0x7fa3fc0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa40373e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc15b490 con 0x7fa3fc0cd390 2024-03-08T08:31:44.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa40373e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0cd770 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa40373e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc15b770 con 0x7fa3fc0ceba0 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa400cd9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa400cd9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54804/0 (socket says 172.21.15.12:54804) 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa400cd9640 1 -- 172.21.15.12:0/494555566 learned_addr learned my addr 172.21.15.12:0/494555566 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.744+0000 7fa4014da640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa3fc0cd390 0x7fa3fc06eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37058/0 (socket says 172.21.15.12:37058) 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa401cdb640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 0x7fa3fc0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa400cd9640 1 -- 172.21.15.12:0/494555566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 msgr2=0x7fa3fc0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa400cd9640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 0x7fa3fc0cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa400cd9640 1 -- 172.21.15.12:0/494555566 >> v1:172.21.15.12:6789/0 conn(0x7fa3fc0cd390 legacy=0x7fa3fc06eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa400cd9640 1 -- 172.21.15.12:0/494555566 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc15ba50 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa401cdb640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 0x7fa3fc0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa400cd9640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009870 tx=0x7fa3f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca56702e38675755 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.745+0000 7fa3fbfff640 1 -- 172.21.15.12:0/494555566 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0034070 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa3fbfff640 1 -- 172.21.15.12:0/494555566 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f0030840 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa3fbfff640 1 -- 172.21.15.12:0/494555566 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0030c10 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 -- 172.21.15.12:0/494555566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc15aeb0 msgr2=0x7fa3fc0b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009870 tx=0x7fa3f002b980 comp rx=0 tx=0).stop 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 -- 172.21.15.12:0/494555566 shutdown_connections 2024-03-08T08:31:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc0ceba0 0x7fa3fc0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 --2- 172.21.15.12:0/494555566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc15aeb0 0x7fa3fc0b3220 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 -- 172.21.15.12:0/494555566 >> 172.21.15.12:0/494555566 conn(0x7fa3fc06abd0 msgr2=0x7fa3fc06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 -- 172.21.15.12:0/494555566 shutdown_connections 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.746+0000 7fa40373e640 1 -- 172.21.15.12:0/494555566 wait complete. 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.747+0000 7fa40373e640 1 Processor -- start 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.747+0000 7fa40373e640 1 -- start start 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 0x7fa3fc0b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3fc0ceba0 0x7fa3fc1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc15aeb0 0x7fa3fc145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0b3760 con 0x7fa3fc0cd390 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0cf550 con 0x7fa3fc0ceba0 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa40373e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0b38e0 con 0x7fa3fc15aeb0 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 0x7fa3fc0b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 0x7fa3fc0b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54812/0 (socket says 172.21.15.12:54812) 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 -- 172.21.15.12:0/27381305 learned_addr learned my addr 172.21.15.12:0/27381305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa400cd9640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3fc0ceba0 0x7fa3fc1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 -- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc15aeb0 msgr2=0x7fa3fc145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa401cdb640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc15aeb0 0x7fa3fc145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3fc15aeb0 0x7fa3fc145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 -- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3fc0ceba0 msgr2=0x7fa3fc1537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.748+0000 7fa4014da640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3fc0ceba0 0x7fa3fc1537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa4014da640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0a85d0 con 0x7fa3fc0cd390 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa400cd9640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3fc0ceba0 0x7fa3fc1537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa4014da640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 0x7fa3fc0b0560 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c3f0 tx=0x7fa3f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40074e0 con 0x7fa3fc0cd390 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4007680 con 0x7fa3fc0cd390 2024-03-08T08:31:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4004a00 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc0a88b0 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.749+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc0a8e70 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.750+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3fc0749c0 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.750+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f401a020 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa3f9ffb640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3f40895d0 con 0x7fa3fc0cd390 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa400cd9640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa3f9ffb640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7fa3e8071390 0x7fa3e80737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3f400bf30 con 0x7fa3e8071390 2024-03-08T08:31:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f4090050 con 0x7fa3fc0cd390 2024-03-08T08:31:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa401cdb640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7fa3e8071390 0x7fa3e80737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.751+0000 7fa400cd9640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0036000 tx=0x7fa3f003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.752+0000 7fa401cdb640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7fa3e8071390 0x7fa3e80737b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.752+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== osd.7 v2:172.21.15.156:6808/1139145947 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3f400bf30 con 0x7fa3e8071390 2024-03-08T08:31:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.766+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 --> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa3fc0af7e0 con 0x7fa3e8071390 2024-03-08T08:31:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.766+0000 7fa3f9ffb640 1 -- 172.21.15.12:0/27381305 <== osd.7 v2:172.21.15.156:6808/1139145947 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa3fc0af7e0 con 0x7fa3e8071390 2024-03-08T08:31:44.768 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-08T08:31:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-08T08:31:44.768 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-08T08:31:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7fa3e8071390 msgr2=0x7fa3e80737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.156:6808/1139145947,v1:172.21.15.156:6809/1139145947] conn(0x7fa3e8071390 0x7fa3e80737b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa3e806b4d0 msgr2=0x7fa3e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0036000 tx=0x7fa3f003b040 comp rx=0 tx=0).stop 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 msgr2=0x7fa3fc0b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 --2- 172.21.15.12:0/27381305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3fc0cd390 0x7fa3fc0b0560 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c3f0 tx=0x7fa3f400c8b0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa401cdb640 1 -- 172.21.15.12:0/27381305 reap_dead start 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 shutdown_connections 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 >> 172.21.15.12:0/27381305 conn(0x7fa3fc06abd0 msgr2=0x7fa3fc0b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 shutdown_connections 2024-03-08T08:31:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.767+0000 7fa40373e640 1 -- 172.21.15.12:0/27381305 wait complete. 2024-03-08T08:31:44.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-08T08:31:44.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:31:44.823 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:31:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.880+0000 7f3ed2a84640 1 Processor -- start 2024-03-08T08:31:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.880+0000 7f3ed2a84640 1 -- start start 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed2a84640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ecc15dbc0 0x7f3ecc0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed2a84640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ecc0d2050 0x7f3ecc0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed2a84640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc0b7540 con 0x7f3ecc0b7080 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed2a84640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b8150 con 0x7f3ecc15dbc0 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed2a84640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b7820 con 0x7f3ecc0d2050 2024-03-08T08:31:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed1021640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ecc0b7080 0x7f3ecc0b85e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37064/0 (socket says 172.21.15.12:37064) 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed1021640 1 -- 172.21.15.12:0/2580672433 learned_addr learned my addr 172.21.15.12:0/2580672433 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ed0820640 1 --2- 172.21.15.12:0/2580672433 >> v2:172.21.15.156:3300/0 conn(0x7f3ecc15dbc0 0x7f3ecc0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ecc0d2050 0x7f3ecc0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.881+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235734985 0 0) 0x7f3ecc0b7540 con 0x7f3ecc0b7080 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb4003520 con 0x7f3ecc0b7080 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1964533608 0 0) 0x7f3eb4003520 con 0x7f3ecc0b7080 2024-03-08T08:31:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ecc0d2050 msgr2=0x7f3ecc0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ecc0d2050 0x7f3ecc0b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 >> v2:172.21.15.156:3300/0 conn(0x7f3ecc15dbc0 msgr2=0x7f3ecc0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 --2- 172.21.15.12:0/2580672433 >> v2:172.21.15.156:3300/0 conn(0x7f3ecc15dbc0 0x7f3ecc0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0b7b00 con 0x7f3ecc0b7080 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3740474727 0 0) 0x7f3ebc001670 con 0x7f3ecc0b7080 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 >> v1:172.21.15.12:6789/0 conn(0x7f3ecc0b7080 legacy=0x7f3ecc0b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb40039b0 0x7f3eb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb40051c0 0x7f3eb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b8150 con 0x7f3eb4004800 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b7820 con 0x7f3eb40039b0 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb400bf00 con 0x7f3eb40051c0 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ed0820640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb40039b0 0x7f3eb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ed1021640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb40051c0 0x7f3eb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 -- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb40051c0 msgr2=0x7f3eb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb40051c0 0x7f3eb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 -- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb40039b0 msgr2=0x7f3eb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb40039b0 0x7f3eb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.882+0000 7f3ecbfff640 1 -- 172.21.15.12:0/2580672433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4003520 con 0x7f3eb4004800 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.883+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009ac0 tx=0x7f3eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc0e3efa46622ef server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.883+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0039070 con 0x7f3eb4004800 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.883+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec002c6b0 con 0x7f3eb4004800 2024-03-08T08:31:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.883+0000 7f3ecb7fe640 1 -- 172.21.15.12:0/2580672433 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0009690 con 0x7f3eb4004800 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 -- 172.21.15.12:0/2580672433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3eb4004800 msgr2=0x7f3eb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/2580672433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009ac0 tx=0x7f3eb4005830 comp rx=0 tx=0).stop 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed1021640 1 -- 172.21.15.12:0/2580672433 reap_dead start 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 -- 172.21.15.12:0/2580672433 shutdown_connections 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 -- 172.21.15.12:0/2580672433 >> 172.21.15.12:0/2580672433 conn(0x7f3ecc0726f0 msgr2=0x7f3ecc072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 -- 172.21.15.12:0/2580672433 shutdown_connections 2024-03-08T08:31:44.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.884+0000 7f3ed2a84640 1 -- 172.21.15.12:0/2580672433 wait complete. 2024-03-08T08:31:44.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.885+0000 7f3ed2a84640 1 Processor -- start 2024-03-08T08:31:44.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.885+0000 7f3ed2a84640 1 -- start start 2024-03-08T08:31:44.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb4004800 0x7f3ecc1e6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ecc1e7b00 0x7f3ecc0c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b8cf0 con 0x7f3ecc14edd0 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc15dfa0 con 0x7f3ecc1e7b00 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b8e70 con 0x7f3eb4004800 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54840/0 (socket says 172.21.15.12:54840) 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 -- 172.21.15.12:0/1991937216 learned_addr learned my addr 172.21.15.12:0/1991937216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ecbfff640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ecc1e7b00 0x7f3ecc0c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed0820640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb4004800 0x7f3ecc1e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 -- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb4004800 msgr2=0x7f3ecc1e6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb4004800 0x7f3ecc1e6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 -- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ecc1e7b00 msgr2=0x7f3ecc0c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ecc1e7b00 0x7f3ecc0c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 -- 172.21.15.12:0/1991937216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc1e6a60 con 0x7f3ecc14edd0 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed1021640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc001cf0 tx=0x7f3ebc00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc00c610 con 0x7f3ecc14edd0 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc018070 con 0x7f3ecc14edd0 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc1e6d40 con 0x7f3ecc14edd0 2024-03-08T08:31:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc00ca10 con 0x7f3ecc14edd0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.886+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc1e7300 con 0x7f3ecc14edd0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.887+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc0749c0 con 0x7f3ecc14edd0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.888+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ebc00cbb0 con 0x7f3ecc14edd0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.888+0000 7f3ec97fa640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.888+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3ebc089b00 con 0x7f3ecc14edd0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.888+0000 7f3ed0820640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:44.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.889+0000 7f3ed0820640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 0x7f3eb806d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc160b80 tx=0x7f3ec400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:44.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:44.893+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ebc0586c0 con 0x7f3ecc14edd0 2024-03-08T08:31:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.075+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ecc0ca430 con 0x7f3eb806b4d0 2024-03-08T08:31:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.075+0000 7f3ec97fa640 1 -- 172.21.15.12:0/1991937216 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3ecc0ca430 con 0x7f3eb806b4d0 2024-03-08T08:31:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 7 to deep-scrub 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 msgr2=0x7f3eb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 0x7f3eb806d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc160b80 tx=0x7f3ec400b040 comp rx=0 tx=0).stop 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 msgr2=0x7f3ecc0b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc001cf0 tx=0x7f3ebc00eb90 comp rx=0 tx=0).stop 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 shutdown_connections 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc14edd0 0x7f3ecc0b4950 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ecc1e7b00 0x7f3ecc0c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 --2- 172.21.15.12:0/1991937216 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3eb4004800 0x7f3ecc1e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 >> 172.21.15.12:0/1991937216 conn(0x7f3ecc0726f0 msgr2=0x7f3ecc15d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:45.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 shutdown_connections 2024-03-08T08:31:45.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.077+0000 7f3ed2a84640 1 -- 172.21.15.12:0/1991937216 wait complete. 2024-03-08T08:31:45.089 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:31:45.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.229+0000 7f90d4eeb640 1 Processor -- start 2024-03-08T08:31:45.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.229+0000 7f90d4eeb640 1 -- start start 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.230+0000 7f90d4eeb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 0x7f90d015f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.230+0000 7f90d4eeb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90d00b14d0 0x7f90d00b1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.230+0000 7f90d4eeb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d00b2df0 con 0x7f90d00b2830 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.230+0000 7f90d4eeb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d0099b90 con 0x7f90d00b14d0 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.230+0000 7f90d4eeb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00cff50 con 0x7f90d015f200 2024-03-08T08:31:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90d00b14d0 0x7f90d00b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90d00b14d0 0x7f90d00b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54842/0 (socket says 172.21.15.12:54842) 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 -- 172.21.15.12:0/4151822677 learned_addr learned my addr 172.21.15.12:0/4151822677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 -- 172.21.15.12:0/4151822677 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 msgr2=0x7f90d015f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ce59c640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 0x7f90d015f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 0x7f90d015f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 -- 172.21.15.12:0/4151822677 >> v1:172.21.15.156:6789/0 conn(0x7f90d00b2830 legacy=0x7f90d015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ced9d640 1 -- 172.21.15.12:0/4151822677 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00d01a0 con 0x7f90d00b14d0 2024-03-08T08:31:45.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.231+0000 7f90ce59c640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 0x7f90d015f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:31:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90ced9d640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b14d0 0x7f90d00b1920 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009e50 tx=0x7f90c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d8d4df719da62e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90cd59a640 1 -- 172.21.15.12:0/4151822677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c002ca70 con 0x7f90d00b14d0 2024-03-08T08:31:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90cd59a640 1 -- 172.21.15.12:0/4151822677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c002cc10 con 0x7f90d00b14d0 2024-03-08T08:31:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90cd59a640 1 -- 172.21.15.12:0/4151822677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c00343f0 con 0x7f90d00b14d0 2024-03-08T08:31:45.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 -- 172.21.15.12:0/4151822677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b14d0 msgr2=0x7f90d00b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b14d0 0x7f90d00b1920 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009e50 tx=0x7f90c002bed0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 -- 172.21.15.12:0/4151822677 shutdown_connections 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b14d0 0x7f90d00b1920 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/4151822677 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d015f200 0x7f90d015f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 -- 172.21.15.12:0/4151822677 >> 172.21.15.12:0/4151822677 conn(0x7f90d00b3460 msgr2=0x7f90d00b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 -- 172.21.15.12:0/4151822677 shutdown_connections 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.232+0000 7f90d4eeb640 1 -- 172.21.15.12:0/4151822677 wait complete. 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 Processor -- start 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 -- start start 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d00b14d0 0x7f90d0152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d015f200 0x7f90d00b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d015f5e0 con 0x7f90d00b2830 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00d07f0 con 0x7f90d015f200 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.234+0000 7f90d4eeb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00cd720 con 0x7f90d00b14d0 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54854/0 (socket says 172.21.15.12:54854) 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 -- 172.21.15.12:0/3283092511 learned_addr learned my addr 172.21.15.12:0/3283092511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90ced9d640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d015f200 0x7f90d00b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90ce59c640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d00b14d0 0x7f90d0152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 -- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d00b14d0 msgr2=0x7f90d0152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d00b14d0 0x7f90d0152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 -- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d015f200 msgr2=0x7f90d00b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d015f200 0x7f90d00b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 -- 172.21.15.12:0/3283092511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00b5210 con 0x7f90d00b2830 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90cdd9b640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c430 tx=0x7f90c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4011790 con 0x7f90d00b2830 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4017070 con 0x7f90d00b2830 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4011bb0 con 0x7f90d00b2830 2024-03-08T08:31:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00b54f0 con 0x7f90d00b2830 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.235+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d00b5ab0 con 0x7f90d00b2830 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.236+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00749c0 con 0x7f90d00b2830 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.237+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90c401c460 con 0x7f90d00b2830 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.237+0000 7f90beffd640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 0x7f90b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.237+0000 7f90ce59c640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 0x7f90b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:31:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.237+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f90c4089b70 con 0x7f90d00b2830 2024-03-08T08:31:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.238+0000 7f90ce59c640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 0x7f90b006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90d00b25f0 tx=0x7f90c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:31:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.242+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f90c4058730 con 0x7f90d00b2830 2024-03-08T08:31:45.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.419+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90d00cdd80 con 0x7f90b006b500 2024-03-08T08:31:45.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.419+0000 7f90beffd640 1 -- 172.21.15.12:0/3283092511 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13405 (secure 0 0 0) 0x7f90d00cdd80 con 0x7f90b006b500 2024-03-08T08:31:45.421 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:31:45.421 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.421+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 msgr2=0x7f90b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.421+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 0x7f90b006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90d00b25f0 tx=0x7f90c8008040 comp rx=0 tx=0).stop 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.421+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 msgr2=0x7f90d00b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.421+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c430 tx=0x7f90c400c8f0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 shutdown_connections 2024-03-08T08:31:45.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f90b006b500 0x7f90b006d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d015f200 0x7f90d00b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90d00b2830 0x7f90d00b62b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 --2- 172.21.15.12:0/3283092511 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90d00b14d0 0x7f90d0152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 >> 172.21.15.12:0/3283092511 conn(0x7f90d00b3460 msgr2=0x7f90d00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 shutdown_connections 2024-03-08T08:31:45.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:31:45.422+0000 7f90d4eeb640 1 -- 172.21.15.12:0/3283092511 wait complete. 2024-03-08T08:31:45.433 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-03-08T08:31:45.026868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":258392,"kb_used_data":4776,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716328,"statfs":{"total":773094113280,"available":772829519872,"internally_reserved":0,"allocated":4890624,"data_stored":2656936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2890,"num_objects":-1000,"num_object_clones":0,"num_object_copies":-2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1000,"num_write_kb":-1000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-64,"num_bytes_recovered":-187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.799715"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-08T08:31:37.416291+0000","last_change":"2024-03-08T08:29:00.157336+0000","last_active":"2024-03-08T08:31:37.416291+0000","last_peered":"2024-03-08T08:31:37.416291+0000","last_clean":"2024-03-08T08:31:37.416291+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:31:37.416291+0000","last_undegraded":"2024-03-08T08:31:37.416291+0000","last_fullsized":"2024-03-08T08:31:37.416291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T08:28:59.148406+0000","last_clean_scrub_stamp":"2024-03-08T08:28:59.148406+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T10:55:00.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509477,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1056768,"data_stored":774893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35800,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335472,"statfs":{"total":96636764160,"available":96599523328,"internally_reserved":0,"allocated":1056768,"data_stored":774893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39952,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:31:45.437 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-08T08:28:59.148406+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=8, tm_hour=8, tm_min=28, tm_sec=59, tm_wday=4, tm_yday=68, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=8, tm_hour=8, tm_min=31, tm_sec=40, tm_wday=4, tm_yday=68, tm_isdst=0) 2024-03-08T08:31:45.437 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-08T08:32:05.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T08:32:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.549+0000 7f8f598d0640 1 Processor -- start 2024-03-08T08:32:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f598d0640 1 -- start start 2024-03-08T08:32:05.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f598d0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8f540d3010 0x7f8f540d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f598d0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540d39d0 con 0x7f8f54163030 2024-03-08T08:32:05.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f598d0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540d3c90 con 0x7f8f540d0ef0 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f598d0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f541634a0 con 0x7f8f540d3010 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f537fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8f54163030 0x7f8f5415f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48300/0 (socket says 172.21.15.12:48300) 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.550+0000 7f8f537fe640 1 -- 172.21.15.12:0/266484314 learned_addr learned my addr 172.21.15.12:0/266484314 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/266484314 >> v2:172.21.15.156:3300/0 conn(0x7f8f540d3010 0x7f8f540d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973173280 0 0) 0x7f8f540d39d0 con 0x7f8f54163030 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c003520 con 0x7f8f54163030 2024-03-08T08:32:05.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173941753 0 0) 0x7f8f540d3c90 con 0x7f8f540d0ef0 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f540d39d0 con 0x7f8f540d0ef0 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f52ffd640 1 -- 172.21.15.12:0/266484314 >> v1:172.21.15.156:6789/0 conn(0x7f8f540d0ef0 legacy=0x7f8f54162920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f52ffd640 1 -- 172.21.15.12:0/266484314 >> v1:172.21.15.12:6789/0 conn(0x7f8f54163030 legacy=0x7f8f5415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583692184 0 0) 0x7f8f3c003520 con 0x7f8f54163030 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.551+0000 7f8f52ffd640 1 -- 172.21.15.12:0/266484314 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540d3f70 con 0x7f8f540d3010 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.552+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/266484314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540d09b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f440097d0 tx=0x7f8f4402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25102ef27eb452ea server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.552+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44038070 con 0x7f8f540d3010 2024-03-08T08:32:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.552+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44031500 con 0x7f8f540d3010 2024-03-08T08:32:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.552+0000 7f8f51ffb640 1 -- 172.21.15.12:0/266484314 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0019f0 con 0x7f8f540d3010 2024-03-08T08:32:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 -- 172.21.15.12:0/266484314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 msgr2=0x7f8f540d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 --2- 172.21.15.12:0/266484314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540d09b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f440097d0 tx=0x7f8f4402b820 comp rx=0 tx=0).stop 2024-03-08T08:32:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 -- 172.21.15.12:0/266484314 shutdown_connections 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 --2- 172.21.15.12:0/266484314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540d09b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 -- 172.21.15.12:0/266484314 >> 172.21.15.12:0/266484314 conn(0x7f8f54072720 msgr2=0x7f8f54072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 -- 172.21.15.12:0/266484314 shutdown_connections 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.553+0000 7f8f598d0640 1 -- 172.21.15.12:0/266484314 wait complete. 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.554+0000 7f8f598d0640 1 Processor -- start 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.554+0000 7f8f598d0640 1 -- start start 2024-03-08T08:32:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 0x7f8f540c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d33f0 con 0x7f8f54163030 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d4530 con 0x7f8f540d3010 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f598d0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d3570 con 0x7f8f540d0ef0 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57078/0 (socket says 172.21.15.12:57078) 2024-03-08T08:32:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 -- 172.21.15.12:0/251305446 learned_addr learned my addr 172.21.15.12:0/251305446 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 0x7f8f540c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f527fc640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 -- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 msgr2=0x7f8f540c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 0x7f8f540c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 -- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 msgr2=0x7f8f540b75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540b75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f537fe640 1 -- 172.21.15.12:0/251305446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54156f90 con 0x7f8f54163030 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.555+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 0x7f8f540c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f537fe640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00cbe0 tx=0x7f8f4c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c017070 con 0x7f8f54163030 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c00ad40 con 0x7f8f54163030 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f540c7ca0 con 0x7f8f54163030 2024-03-08T08:32:05.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c012600 con 0x7f8f54163030 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.556+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f540c8260 con 0x7f8f54163030 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.557+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f540749c0 con 0x7f8f54163030 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.557+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f4c01a070 con 0x7f8f54163030 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.557+0000 7f8f3bfff640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 0x7f8f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.558+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8f4c089c00 con 0x7f8f54163030 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.558+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 0x7f8f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.558+0000 7f8f52ffd640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 0x7f8f4006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f44035000 tx=0x7f8f44008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.562+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8f4c090080 con 0x7f8f54163030 2024-03-08T08:32:05.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.741+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 --> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f540cdb60 con 0x7f8f4006b4d0 2024-03-08T08:32:05.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.741+0000 7f8f3bfff640 1 -- 172.21.15.12:0/251305446 <== mgr.4110 v2:172.21.15.12:6832/2068370000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13359 (secure 0 0 0) 0x7f8f540cdb60 con 0x7f8f4006b4d0 2024-03-08T08:32:05.743 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:32:05.743 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-08T08:32:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 msgr2=0x7f8f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 0x7f8f4006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f44035000 tx=0x7f8f44008030 comp rx=0 tx=0).stop 2024-03-08T08:32:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 msgr2=0x7f8f54156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00cbe0 tx=0x7f8f4c00a430 comp rx=0 tx=0).stop 2024-03-08T08:32:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 shutdown_connections 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7f8f4006b4d0 0x7f8f4006d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f54163030 0x7f8f54156a50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f540d3010 0x7f8f540b75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 --2- 172.21.15.12:0/251305446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f540d0ef0 0x7f8f540c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 >> 172.21.15.12:0/251305446 conn(0x7f8f54072720 msgr2=0x7f8f540cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 shutdown_connections 2024-03-08T08:32:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.743+0000 7f8f598d0640 1 -- 172.21.15.12:0/251305446 wait complete. 2024-03-08T08:32:05.755 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-03-08T08:32:05.030733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":258384,"kb_used_data":4776,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716336,"statfs":{"total":773094113280,"available":772829528064,"internally_reserved":0,"allocated":4890624,"data_stored":2656936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002255"},"pg_stats":[{"pgid":"1.0","version":"40'119","reported_seq":217,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-08T08:31:43.064368+0000","last_change":"2024-03-08T08:31:43.064398+0000","last_active":"2024-03-08T08:31:43.064368+0000","last_peered":"2024-03-08T08:31:43.064368+0000","last_clean":"2024-03-08T08:31:43.064368+0000","last_became_active":"2024-03-08T08:29:00.156490+0000","last_became_peered":"2024-03-08T08:29:00.156490+0000","last_unstale":"2024-03-08T08:31:43.064368+0000","last_undegraded":"2024-03-08T08:31:43.064368+0000","last_fullsized":"2024-03-08T08:31:43.064368+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'119","last_scrub_stamp":"2024-03-08T08:31:43.064286+0000","last_deep_scrub":"40'119","last_deep_scrub_stamp":"2024-03-08T08:31:43.064286+0000","last_clean_scrub_stamp":"2024-03-08T08:31:43.064286+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T14:03:33.495306+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339632,"statfs":{"total":96636764160,"available":96603783168,"internally_reserved":0,"allocated":1056768,"data_stored":774893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35800,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335472,"statfs":{"total":96636764160,"available":96599523328,"internally_reserved":0,"allocated":1056768,"data_stored":774893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39952,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":184525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T08:32:05.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-08T08:32:05.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.856+0000 7fd06405c640 1 Processor -- start 2024-03-08T08:32:05.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 -- start start 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd05c06e200 0x7fd05c06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd05c06e5e0 0x7fd05c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c15aa70 con 0x7fd05c06e5e0 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c15ad50 con 0x7fd05c06e200 2024-03-08T08:32:05.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd06405c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c15b030 con 0x7fd05c0aef80 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.857+0000 7fd0625f9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd05c06e5e0 0x7fd05c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd05c06e5e0 0x7fd05c15a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57090/0 (socket says 172.21.15.12:57090) 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 -- 172.21.15.12:0/2241263051 learned_addr learned my addr 172.21.15.12:0/2241263051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0615f7640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd061df8640 1 --2- 172.21.15.12:0/2241263051 >> v2:172.21.15.156:3300/0 conn(0x7fd05c06e200 0x7fd05c06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 -- 172.21.15.12:0/2241263051 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 msgr2=0x7fd05c159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:32:05.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 -- 172.21.15.12:0/2241263051 >> v2:172.21.15.156:3300/0 conn(0x7fd05c06e200 msgr2=0x7fd05c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 --2- 172.21.15.12:0/2241263051 >> v2:172.21.15.156:3300/0 conn(0x7fd05c06e200 0x7fd05c06ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.858+0000 7fd0625f9640 1 -- 172.21.15.12:0/2241263051 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c06f350 con 0x7fd05c06e5e0 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd0625f9640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e5e0 0x7fd05c15a060 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd05c072850 tx=0x7fd05402cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2f66c3d34d70f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd060df6640 1 -- 172.21.15.12:0/2241263051 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05402da60 con 0x7fd05c06e5e0 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd060df6640 1 -- 172.21.15.12:0/2241263051 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05402dc00 con 0x7fd05c06e5e0 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd060df6640 1 -- 172.21.15.12:0/2241263051 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0540353f0 con 0x7fd05c06e5e0 2024-03-08T08:32:05.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 -- 172.21.15.12:0/2241263051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e5e0 msgr2=0x7fd05c15a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e5e0 0x7fd05c15a060 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd05c072850 tx=0x7fd05402cec0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 -- 172.21.15.12:0/2241263051 shutdown_connections 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e5e0 0x7fd05c15a060 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 --2- 172.21.15.12:0/2241263051 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 --2- 172.21.15.12:0/2241263051 >> v2:172.21.15.156:3300/0 conn(0x7fd05c06e200 0x7fd05c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 -- 172.21.15.12:0/2241263051 >> 172.21.15.12:0/2241263051 conn(0x7fd05c06abd0 msgr2=0x7fd05c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:05.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 -- 172.21.15.12:0/2241263051 shutdown_connections 2024-03-08T08:32:05.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.859+0000 7fd06405c640 1 -- 172.21.15.12:0/2241263051 wait complete. 2024-03-08T08:32:05.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.860+0000 7fd06405c640 1 Processor -- start 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 -- start start 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 0x7fd05c153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c1f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c06d700 con 0x7fd05c06e200 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c15a7b0 con 0x7fd05c06e5e0 2024-03-08T08:32:05.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd06405c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c06d880 con 0x7fd05c0aef80 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd061df8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd061df8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57092/0 (socket says 172.21.15.12:57092) 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.861+0000 7fd061df8640 1 -- 172.21.15.12:0/1042476471 learned_addr learned my addr 172.21.15.12:0/1042476471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0625f9640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c1f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0615f7640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 0x7fd05c153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 -- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 msgr2=0x7fd05c1f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c1f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 -- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 msgr2=0x7fd05c153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 0x7fd05c153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 -- 172.21.15.12:0/1042476471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c1f74a0 con 0x7fd05c06e200 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0625f9640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c1f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0615f7640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 0x7fd05c153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd061df8640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd05800e890 tx=0x7fd05800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd058017070 con 0x7fd05c06e200 2024-03-08T08:32:05.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05800c940 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c1f7770 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05800cd10 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.862+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c1f7d30 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.863+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c0749c0 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.863+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd05801a070 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.864+0000 7fd0527fc640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.864+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd058089ec0 con 0x7fd05c06e200 2024-03-08T08:32:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.864+0000 7fd0615f7640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.864+0000 7fd0615f7640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 0x7fd04006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0b4bf0 tx=0x7fd04c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:05.868+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd058059040 con 0x7fd05c06e200 2024-03-08T08:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.048+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fd05c072950 con 0x7fd05c06e200 2024-03-08T08:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.050+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) v1 ==== 123+0+0 (secure 0 0 0) 0x7fd058058940 con 0x7fd05c06e200 2024-03-08T08:32:06.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.052+0000 7fd0527fc640 1 -- 172.21.15.12:0/1042476471 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd05805ca90 con 0x7fd05c06e200 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 msgr2=0x7fd04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 0x7fd04006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0b4bf0 tx=0x7fd04c008040 comp rx=0 tx=0).stop 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 msgr2=0x7fd05c1f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd05800e890 tx=0x7fd05800ed50 comp rx=0 tx=0).stop 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 shutdown_connections 2024-03-08T08:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd05c0aef80 0x7fd05c1f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd05c06e5e0 0x7fd05c153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 --2- 172.21.15.12:0/1042476471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c06e200 0x7fd05c1f6f60 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 >> 172.21.15.12:0/1042476471 conn(0x7fd05c06abd0 msgr2=0x7fd05c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 shutdown_connections 2024-03-08T08:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.053+0000 7fd06405c640 1 -- 172.21.15.12:0/1042476471 wait complete. 2024-03-08T08:32:06.065 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-08T08:32:06.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T08:32:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.165+0000 7fd043235640 1 Processor -- start 2024-03-08T08:32:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.165+0000 7fd043235640 1 -- start start 2024-03-08T08:32:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd043235640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd043235640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd03c0d0f40 0x7fd03c0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd043235640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c15e1a0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd043235640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03c0b9030 con 0x7fd03c0d0f40 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd043235640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d13a0 con 0x7fd03c15ddc0 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd03bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd03c0d0f40 0x7fd03c0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd0417d2640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd03c0d26f0 0x7fd03c0d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48314/0 (socket says 172.21.15.12:48314) 2024-03-08T08:32:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.166+0000 7fd0417d2640 1 -- 172.21.15.12:0/907835232 learned_addr learned my addr 172.21.15.12:0/907835232 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd040fd1640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03bfff640 1 -- 172.21.15.12:0/907835232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 msgr2=0x7fd03c0d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03bfff640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03bfff640 1 -- 172.21.15.12:0/907835232 >> v1:172.21.15.12:6789/0 conn(0x7fd03c0d26f0 legacy=0x7fd03c0d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03bfff640 1 -- 172.21.15.12:0/907835232 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0b8790 con 0x7fd03c0d0f40 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03b7fe640 1 -- 172.21.15.12:0/907835232 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680796872 0 0) 0x7fd03c15e1a0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.167+0000 7fd03bfff640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0d21b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd0240097a0 tx=0x7fd02402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1007278fcb5071b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd03b7fe640 1 -- 172.21.15.12:0/907835232 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024035020 con 0x7fd03c0d0f40 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd03b7fe640 1 -- 172.21.15.12:0/907835232 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd024031830 con 0x7fd03c0d0f40 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd03b7fe640 1 -- 172.21.15.12:0/907835232 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024031c20 con 0x7fd03c0d0f40 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 -- 172.21.15.12:0/907835232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d0f40 msgr2=0x7fd03c0d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0d21b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd0240097a0 tx=0x7fd02402b8d0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 -- 172.21.15.12:0/907835232 shutdown_connections 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0d21b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 --2- 172.21.15.12:0/907835232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 -- 172.21.15.12:0/907835232 >> 172.21.15.12:0/907835232 conn(0x7fd03c072720 msgr2=0x7fd03c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:06.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 -- 172.21.15.12:0/907835232 shutdown_connections 2024-03-08T08:32:06.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.168+0000 7fd043235640 1 -- 172.21.15.12:0/907835232 wait complete. 2024-03-08T08:32:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.169+0000 7fd043235640 1 Processor -- start 2024-03-08T08:32:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.169+0000 7fd043235640 1 -- start start 2024-03-08T08:32:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b8d50 con 0x7fd03c0d26f0 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d1d20 con 0x7fd03c0d0f40 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd043235640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b9980 con 0x7fd03c15ddc0 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd03bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd03bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57114/0 (socket says 172.21.15.12:57114) 2024-03-08T08:32:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd03bfff640 1 -- 172.21.15.12:0/3047943988 learned_addr learned my addr 172.21.15.12:0/3047943988 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd0417d2640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.170+0000 7fd040fd1640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 -- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 msgr2=0x7fd03c0ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 -- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 msgr2=0x7fd03c0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 -- 172.21.15.12:0/3047943988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c1f0f90 con 0x7fd03c0d26f0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd0417d2640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd040fd1640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd03bfff640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd0240098c0 tx=0x7fd0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024041070 con 0x7fd03c0d26f0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd02403a950 con 0x7fd03c0d26f0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c0aabb0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024035040 con 0x7fd03c0d26f0 2024-03-08T08:32:06.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.171+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c0ab170 con 0x7fd03c0d26f0 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.172+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c0749c0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.172+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd02404c060 con 0x7fd03c0d26f0 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.173+0000 7fd0397fa640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.173+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd0240ab9d0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.173+0000 7fd040fd1640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T08:32:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.173+0000 7fd040fd1640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 0x7fd01c06d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0cd230 tx=0x7fd03400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T08:32:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.177+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd02407ab50 con 0x7fd03c0d26f0 2024-03-08T08:32:06.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.393+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd03c15d1c0 con 0x7fd03c0d26f0 2024-03-08T08:32:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.394+0000 7fd0397fa640 1 -- 172.21.15.12:0/3047943988 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3704 (secure 0 0 0) 0x7fd02407a450 con 0x7fd03c0d26f0 2024-03-08T08:32:06.395 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-08T08:32:06.395 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4697,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/2749785025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2749785025},{"type":"v1","addr":"172.21.15.156:6834","nonce":2749785025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4700,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2735210136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":2735210136},{"type":"v1","addr":"172.21.15.156:6835","nonce":2735210136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1355234906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1355234906},{"type":"v1","addr":"172.21.15.12:6835","nonce":1355234906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/3223799658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":3223799658},{"type":"v1","addr":"172.21.15.12:6837","nonce":3223799658}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2024-03-08T08:32:06.396 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-03-08T08:32:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 msgr2=0x7fd01c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 0x7fd01c06d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0cd230 tx=0x7fd03400b040 comp rx=0 tx=0).stop 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 msgr2=0x7fd03c1f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd0240098c0 tx=0x7fd0240046c0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 shutdown_connections 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:6832/2068370000,v1:172.21.15.12:6833/2068370000] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03c15ddc0 0x7fd03c0ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0d26f0 0x7fd03c1f0a50 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 --2- 172.21.15.12:0/3047943988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd03c0d0f40 0x7fd03c0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T08:32:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 >> 172.21.15.12:0/3047943988 conn(0x7fd03c072720 msgr2=0x7fd03c0d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T08:32:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 shutdown_connections 2024-03-08T08:32:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-08T08:32:06.396+0000 7fd043235640 1 -- 172.21.15.12:0/3047943988 wait complete. 2024-03-08T08:32:06.408 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-08T08:32:06.408 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-08T08:32:06.408 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-08T08:32:06.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.417 INFO:tasks.ceph.mds.a:Stopped 2024-03-08T08:32:06.418 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-08T08:32:06.418 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.467 INFO:tasks.ceph.mds.c:Stopped 2024-03-08T08:32:06.467 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-08T08:32:06.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.479 INFO:tasks.ceph.mds.b:Stopped 2024-03-08T08:32:06.480 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-08T08:32:06.480 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.531 INFO:tasks.ceph.mds.d:Stopped 2024-03-08T08:32:06.531 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-08T08:32:06.532 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-08T08:32:06.532 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.557 INFO:tasks.ceph.osd.0:Stopped 2024-03-08T08:32:06.557 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-08T08:32:06.557 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.623 INFO:tasks.ceph.osd.1:Stopped 2024-03-08T08:32:06.624 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-08T08:32:06.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.697 INFO:tasks.ceph.osd.2:Stopped 2024-03-08T08:32:06.698 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-08T08:32:06.698 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.778 INFO:tasks.ceph.osd.3:Stopped 2024-03-08T08:32:06.779 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-08T08:32:06.779 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.813 INFO:tasks.ceph.osd.4:Stopped 2024-03-08T08:32:06.814 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-08T08:32:06.814 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.882 INFO:tasks.ceph.osd.5:Stopped 2024-03-08T08:32:06.882 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-08T08:32:06.882 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:06.957 INFO:tasks.ceph.osd.6:Stopped 2024-03-08T08:32:06.958 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-08T08:32:06.958 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.028 INFO:tasks.ceph.osd.7:Stopped 2024-03-08T08:32:07.028 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-08T08:32:07.028 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-08T08:32:07.029 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.055 INFO:tasks.ceph.mgr.y:Stopped 2024-03-08T08:32:07.055 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-08T08:32:07.055 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.096 INFO:tasks.ceph.mgr.x:Stopped 2024-03-08T08:32:07.096 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-08T08:32:07.097 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-08T08:32:07.097 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.107 INFO:tasks.ceph.mon.a:Stopped 2024-03-08T08:32:07.108 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-08T08:32:07.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.149 INFO:tasks.ceph.mon.b:Stopped 2024-03-08T08:32:07.149 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-08T08:32:07.149 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T08:32:07.202 INFO:tasks.ceph.mon.c:Stopped 2024-03-08T08:32:07.202 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-08T08:32:07.202 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'missing required features' | head -n 1 2024-03-08T08:32:07.235 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:32:07.235 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-08T08:32:07.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:32:07.425 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-08T08:32:07.509 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:32:07.509 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-08T08:32:07.566 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:32:07.567 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-08T08:32:07.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:32:07.616 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-08T08:32:07.807 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:32:07.807 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-08T08:32:07.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:32:07.870 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-08T08:32:07.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:32:07.920 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-08T08:32:07.967 INFO:tasks.ceph:Archiving mon data... 2024-03-08T08:32:07.968 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/data/mon.a.tgz 2024-03-08T08:32:07.968 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-08T08:32:07.983 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.iDHpDryb9c 2024-03-08T08:32:07.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iDHpDryb9c 2024-03-08T08:32:08.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.iDHpDryb9c 2024-03-08T08:32:08.219 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.iDHpDryb9c is 300KB 2024-03-08T08:32:08.315 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.iDHpDryb9c 2024-03-08T08:32:08.329 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/data/mon.b.tgz 2024-03-08T08:32:08.329 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-08T08:32:08.346 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.r7oTlpNOJB 2024-03-08T08:32:08.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.r7oTlpNOJB 2024-03-08T08:32:08.497 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.r7oTlpNOJB 2024-03-08T08:32:08.582 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.r7oTlpNOJB is 400KB 2024-03-08T08:32:08.678 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.r7oTlpNOJB 2024-03-08T08:32:08.692 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/data/mon.c.tgz 2024-03-08T08:32:08.692 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-08T08:32:08.746 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.02PH9kYQTE 2024-03-08T08:32:08.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.02PH9kYQTE 2024-03-08T08:32:08.885 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.02PH9kYQTE 2024-03-08T08:32:08.970 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.02PH9kYQTE is 400KB 2024-03-08T08:32:09.066 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.02PH9kYQTE 2024-03-08T08:32:09.081 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-08T08:32:09.081 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-08T08:32:09.083 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-08T08:32:09.166 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-08T08:32:09.202 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-08T08:32:09.202 INFO:tasks.ceph:Archiving crash dumps... 2024-03-08T08:32:09.204 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi012/crash 2024-03-08T08:32:09.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-08T08:32:09.234 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi156/crash 2024-03-08T08:32:09.235 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-08T08:32:09.268 INFO:tasks.ceph:Shutting down logrotate 2024-03-08T08:32:09.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:32:09.278 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-08T08:32:09.354 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-08T08:32:09.381 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-08T08:32:09.408 INFO:tasks.ceph:Compressing logs... 2024-03-08T08:32:09.408 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-08T08:32:09.424 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-08T08:32:09.448 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45249.log 2024-03-08T08:32:09.448 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2024-03-08T08:32:09.502 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-08T08:32:09.502 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43673.log 2024-03-08T08:32:09.502 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37569.log 2024-03-08T08:32:09.502 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.43673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37489.log 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37569.log.gz 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37489.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37489.log.gz 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37369.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37529.log 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-03-08T08:32:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37409.log 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.4.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37529.log.gz 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37409.log: 92.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37409.log.gz 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose 93.7% -- /var/log/ceph/ceph-client.admin.37449.log -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-08T08:32:09.504 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37289.log 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.37449.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37289.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37449.log.gz 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37289.log.gz 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42497.log 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-03-08T08:32:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.42497.log: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.1.42497.log.gz 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.d.log: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42950.log 2024-03-08T08:32:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.3.42950.log: 90.3% -- replaced with /var/log/ceph/ceph-client.3.42950.log.gz 2024-03-08T08:32:09.535 INFO:teuthology.orchestra.run.smithi156.stderr: 94.8% -- replaced with /var/log/ceph/ceph-client.1.43673.log.gz 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54378.log 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42675.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.45249.log.gz 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48489.log 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54378.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gz 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-03-08T08:32:09.565 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48489.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54378.log.gz 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48489.log.gz 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47544.log 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51368.log 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51056.log 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47544.log.gz 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49059.log 2024-03-08T08:32:09.566 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51056.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.51368.log.gz 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51934.log 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49059.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51056.log.gz 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53113.log 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49059.log.gz 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51098.log 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51098.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53113.log.gz 2024-03-08T08:32:09.567 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51934.log.gz 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47712.log 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51098.log.gz 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52288.log 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43905.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47712.log.gz 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42990.log 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2024-03-08T08:32:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50019.log 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52288.log.gz 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43635.log 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42990.log.gz 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43158.log 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50537.log 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50019.log.gz 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43158.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43635.log.gz 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 2024-03-08T08:32:09.569 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43158.log.gz 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45081.log 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49893.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.50537.log.gz 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44085.log 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50453.log 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45081.log.gz 2024-03-08T08:32:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44412.log 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44085.log.gz 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49431.log 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50453.log.gz 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53790.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44412.log.gz 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47292.log 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2024-03-08T08:32:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47292.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49431.log.gz 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47292.log.gz 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48017.log 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47166.log 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48017.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2024-03-08T08:32:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44414.log 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48017.log.gz 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54336.log 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47166.log.gz 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48321.log 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54336.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44414.log.gz 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48321.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54336.log.gz 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48243.log 2024-03-08T08:32:09.573 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48321.log.gz 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51735.log 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48243.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44756.log 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48243.log.gz 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.51735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48405.log 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54252.log 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44756.log.gz 2024-03-08T08:32:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50061.log 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49185.log 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9%/var/log/ceph/ceph-client.admin.54252.log: -- replaced with /var/log/ceph/ceph-client.admin.48405.log.gz 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49017.log 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54252.log.gz 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50061.log.gz 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49599.log 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49683.log 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49017.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49185.log.gz 2024-03-08T08:32:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.49599.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49017.log.gz 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49683.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49599.log.gz 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48833.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49683.log.gz 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50103.log 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-03-08T08:32:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52903.log 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44423.log: /var/log/ceph/ceph-client.admin.50103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54294.log 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50103.log.gz 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54438.log 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43719.log 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52903.log.gz 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54438.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54294.log.gz 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47586.log 2024-03-08T08:32:09.577 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54438.log.gz 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53342.log 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43719.log: /var/log/ceph/ceph-client.admin.47586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48717.log 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43719.log.gz 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53342.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47586.log.gz 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52414.log 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53342.log.gz 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43116.log 2024-03-08T08:32:09.578 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48717.log: /var/log/ceph/ceph-client.admin.52414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49263.log 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48717.log.gz 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43074.log 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52414.log.gz 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2%/var/log/ceph/ceph-client.admin.49263.log: -- replaced with /var/log/ceph/ceph-client.admin.43116.log.gz 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44083.log 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49263.log.gz 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42773.log 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43074.log: /var/log/ceph/ceph-client.admin.44083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 2024-03-08T08:32:09.579 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43074.log.gz 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42773.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44083.log.gz 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50831.log 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42773.log.gz 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48075.log 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44619.log 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48075.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50831.log.gz 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42854.log 2024-03-08T08:32:09.580 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44619.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48075.log.gz 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50495.log 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44619.log.gz 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47502.log 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42854.log: /var/log/ceph/ceph-client.admin.50495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53257.log 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42854.log.gz 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47502.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50495.log.gz 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53029.log 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47502.log.gz 2024-03-08T08:32:09.581 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49305.log 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53257.log: /var/log/ceph/ceph-client.admin.53029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46226.log 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53257.log.gz 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53029.log.gz 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47082.log 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43339.log 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46226.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49305.log.gz 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46226.log.gz 2024-03-08T08:32:09.582 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43593.log 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50163.log 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43339.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47082.log.gz 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43339.log.gz 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43593.log.gz 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44086.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-08T08:32:09.583 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50163.log.gz 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37432.log 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51158.log 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.audit.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52618.log 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51158.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37432.log.gz 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51200.log 2024-03-08T08:32:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52618.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51158.log.gz 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46720.log 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5%/var/log/ceph/ceph-client.admin.51200.log: -- replaced with /var/log/ceph/ceph-client.admin.52618.log.gz 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44677.log 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51200.log.gz 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53384.log 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47670.log 2024-03-08T08:32:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44677.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46720.log.gz 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53384.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44677.log.gz 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49767.log 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49389.log 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47670.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46103.log 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47670.log.gz 2024-03-08T08:32:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47933.log 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49389.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49767.log.gz 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.46103.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49389.log.gz 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45039.log 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48615.log 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47933.log: /var/log/ceph/ceph-client.admin.45039.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47933.log.gz 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45039.log.gz 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.43032.log 2024-03-08T08:32:09.587 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.46103.log.gz 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44997.log 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54520.log 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48615.log.gz 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43032.log.gz 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48959.log 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49725.log 2024-03-08T08:32:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54520.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44997.log.gz 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54520.log.gz 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48959.log.gz 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45417.log 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49725.log: /var/log/ceph/ceph-client.admin.53874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53426.log 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49725.log.gz 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45417.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-03-08T08:32:09.589 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54084.log 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45417.log.gz 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53426.log: /var/log/ceph/ceph-client.admin.54000.log 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47849.log 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53426.log.gz 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54084.log.gz 2024-03-08T08:32:09.590 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54000.log.gz 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44419.log 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47849.log.gz 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43246.log: /var/log/ceph/ceph-client.admin.44419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43297.log 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50970.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44419.log.gz 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2024-03-08T08:32:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gz 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43297.log: /var/log/ceph/ceph-client.admin.45543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50349.log 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43297.log.gz 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49143.log 2024-03-08T08:32:09.592 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.50349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48675.log 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49143.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50349.log.gz 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44420.log 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49143.log.gz 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48675.log: -- /var/log/ceph/ceph-client.admin.43677.log 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48675.log.gz 2024-03-08T08:32:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54042.log 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44420.log.gz 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43677.log.gz 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45501.log 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52534.log 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54042.log: /var/log/ceph/ceph-client.admin.45501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51410.log 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54042.log.gz 2024-03-08T08:32:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52534.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46998.log 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45501.log.gz 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51410.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52534.log.gz 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42896.log 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46998.log.gz 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42896.log: gzip -5 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51410.log.gz 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52083.log 2024-03-08T08:32:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45291.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42896.log.gz 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 89.2% 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46872.log 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54210.log 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49347.log 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52083.log.gz 2024-03-08T08:32:09.596 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6%/var/log/ceph/ceph-client.admin.54210.log: -- replaced with /var/log/ceph/ceph-client.admin.46872.log.gz 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53748.log 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54210.log.gz 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47124.log 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49347.log: /var/log/ceph/ceph-client.admin.53748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52819.log 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49347.log.gz 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46762.log 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53748.log.gz 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52819.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47124.log.gz 2024-03-08T08:32:09.597 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52456.log 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52819.log.gz 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.46762.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48363.log 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46762.log.gz 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44087.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52456.log.gz 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49935.log 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48531.log 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48363.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2024-03-08T08:32:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50579.log 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48363.log.gz 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45207.log 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53071.log 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48531.log.gz 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45207.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-03-08T08:32:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53071.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45207.log.gz 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44758.log 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53958.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53071.log.gz 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48159.log 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48573.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44758.log.gz 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48159.log: /var/log/ceph/ceph-client.admin.43761.log 2024-03-08T08:32:09.600 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.47975.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44679.log 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48159.log.gz 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43761.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz -5 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49557.log 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43761.log.gz 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44679.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44079.log 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44679.log.gz 2024-03-08T08:32:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.3.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49557.log.gz 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51326.log 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44079.log.gz 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52576.log 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52330.log 2024-03-08T08:32:09.602 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51326.log.gz 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51892.log 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52576.log.gz 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52330.log.gz 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47334.log 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53197.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51892.log.gz 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2024-03-08T08:32:09.603 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50747.log 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47334.log.gz 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47376.log 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50747.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49851.log 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50747.log.gz 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50621.log 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2024-03-08T08:32:09.604 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47376.log.gz 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53706.log 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49851.log.gz 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43551.log 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50621.log.gz 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53706.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49473.log 2024-03-08T08:32:09.605 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43551.log.gz 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44418.log 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50705.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49473.log.gz 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52372.log 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44417.log 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44418.log.gz 2024-03-08T08:32:09.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51284.log 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44417.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52372.log.gz 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44417.log.gz 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50875.log 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51284.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51284.log.gz 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44422.log 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47418.log 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz/var/log/ceph/ceph-client.admin.44422.log: 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50206.log 87.4% 2024-03-08T08:32:09.607 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44422.log.gz 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47418.log.gz/var/log/ceph/ceph-client.admin.50206.log: 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43467.log 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50206.log.gz 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44082.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31357.log 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43509.log 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43467.log.gz 2024-03-08T08:32:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.tmp-client.admin.31357.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31357.log.gz 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51476.log 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53916.log 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2%/var/log/ceph/ceph-client.admin.51476.log: -- replaced with /var/log/ceph/ceph-client.admin.43509.log.gz 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52702.log 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51476.log.gz 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48201.log 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53916.log: /var/log/ceph/ceph-client.admin.52702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53832.log 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48201.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53916.log.gz 2024-03-08T08:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52702.log.gz 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48201.log.gz 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54126.log 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48917.log 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53832.log: /var/log/ceph/ceph-client.admin.54126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47208.log 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48917.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53832.log.gz 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54126.log.gz 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48917.log.gz 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-08T08:32:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43381.log 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.47208.log.gz 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44081.log 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43381.log: /var/log/ceph/ceph-client.admin.53622.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43381.log.gz 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50248.log 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44081.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44081.log.gz 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52167.log 2024-03-08T08:32:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50248.log: /var/log/ceph/ceph-client.admin.52167.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50248.log.gz 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50919.log 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0%/var/log/ceph/ceph-client.admin.53155.log: -- replaced with /var/log/ceph/ceph-client.admin.52167.log.gz 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54480.log 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47628.log 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50919.log: /var/log/ceph/ceph-client.admin.54480.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50919.log.gz 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54480.log.gz 2024-03-08T08:32:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49809.log 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47628.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47628.log.gz 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46914.log 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49809.log: /var/log/ceph/ceph-client.admin.46914.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49809.log.gz 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46914.log.gz 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48447.log 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.log: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-08T08:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53664.log 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48447.log: /var/log/ceph/ceph-client.admin.53664.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48447.log.gz 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53664.log.gz 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mds.c.log: 92.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50411.log 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44757.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-03-08T08:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43425.log 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50411.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50411.log.gz 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44763.log 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43425.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43425.log.gz 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46598.log 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44763.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44763.log.gz 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46956.log 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.2.46598.log: 90.7% -- replaced with /var/log/ceph/ceph-client.2.46598.log.gz 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52777.log 2024-03-08T08:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46956.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46956.log.gz 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43202.log 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52777.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52777.log.gz 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45375.log 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43202.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43202.log.gz 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49101.log 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45375.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45375.log.gz 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52861.log 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49101.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49101.log.gz 2024-03-08T08:32:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52861.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52861.log.gz 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47460.log 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53580.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45459.log 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47460.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47460.log.gz 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45687.log 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45459.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45459.log.gz 2024-03-08T08:32:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49977.log 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45687.log: 91.5% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45687.log.gz 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44078.log 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51242.log 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49977.log: /var/log/ceph/ceph-client.admin.44078.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49977.log.gz 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50789.log 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44078.log.gz 2024-03-08T08:32:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52945.log 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50789.log: /var/log/ceph/ceph-client.admin.51242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51242.log.gz 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50789.log.gz 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52945.log: -- /var/log/ceph/ceph-client.admin.43803.log 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52945.log.gz 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43863.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48117.log 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47250.log 2024-03-08T08:32:09.619 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43803.log.gz 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49641.log 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47250.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48117.log.gz 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47250.log.gz 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50663.log 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52660.log 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49641.log: /var/log/ceph/ceph-client.admin.50663.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49641.log.gz 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50663.log.gz 2024-03-08T08:32:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47040.log 2024-03-08T08:32:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2024-03-08T08:32:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47040.log: /var/log/ceph/ceph-client.admin.52660.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52660.log.gz 2024-03-08T08:32:09.621 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47040.log.gz 2024-03-08T08:32:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-08T08:32:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47891.log 2024-03-08T08:32:09.622 INFO:teuthology.orchestra.run.smithi156.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-08T08:32:09.622 INFO:teuthology.orchestra.run.smithi156.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-08T08:32:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48875.log: /var/log/ceph/ceph-mds.a.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2024-03-08T08:32:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49515.log 2024-03-08T08:32:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47891.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47891.log.gz 2024-03-08T08:32:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54168.log 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49515.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49515.log.gz 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37682.log 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54168.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54168.log.gz 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54562.log 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37682.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37682.log.gz 2024-03-08T08:32:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54562.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54562.log.gz 2024-03-08T08:32:09.635 INFO:teuthology.orchestra.run.smithi156.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-08T08:32:09.681 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-08T08:32:09.693 INFO:teuthology.orchestra.run.smithi156.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-08T08:32:09.737 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-08T08:32:09.748 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-08T08:32:09.749 INFO:teuthology.orchestra.run.smithi012.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-08T08:32:09.785 INFO:teuthology.orchestra.run.smithi012.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-08T08:32:09.800 INFO:teuthology.orchestra.run.smithi156.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-08T08:32:09.816 INFO:teuthology.orchestra.run.smithi156.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-08T08:32:09.818 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-08T08:32:09.819 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.349s 2024-03-08T08:32:09.819 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m1.241s 2024-03-08T08:32:09.819 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.070s 2024-03-08T08:32:10.033 INFO:teuthology.orchestra.run.smithi012.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.51735.log.gz 2024-03-08T08:32:10.871 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-08T08:32:10.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-03-08T08:32:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m1.437s 2024-03-08T08:32:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m3.179s 2024-03-08T08:32:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.257s 2024-03-08T08:32:10.873 INFO:tasks.ceph:Archiving logs... 2024-03-08T08:32:10.874 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi012/log 2024-03-08T08:32:10.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-08T08:32:11.508 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi156/log 2024-03-08T08:32:11.509 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-08T08:32:11.706 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-08T08:32:11.716 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-08T08:32:11.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-08T08:32:11.719 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-08T08:32:11.809 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-08T08:32:11.818 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-08T08:32:11.819 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T08:32:11.821 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T08:32:11.833 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-08T08:32:11.836 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -880us[ -880us] +/- 141ms 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -124us[ -126us] +/- 76ms 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +634us[ +634us] +/- 77ms 2024-03-08T08:32:11.839 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:32:11.841 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T08:32:11.841 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-08T08:32:11.841 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -869us[ -869us] +/- 141ms 2024-03-08T08:32:11.841 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -106us[ -88us] +/- 76ms 2024-03-08T08:32:11.842 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +649us[ +649us] +/- 77ms 2024-03-08T08:32:11.842 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T08:32:11.843 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-08T08:32:11.914 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-08T08:32:11.915 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-08T08:32:11.994 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-08T08:32:12.071 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-08T08:32:12.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-08T08:32:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:32:12.176 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-08T08:32:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:32:12.208 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-08T08:32:12.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-08T08:32:12.292 INFO:teuthology.task.internal:Duration was 755.369329 seconds 2024-03-08T08:32:12.292 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-08T08:32:12.425 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-08T08:32:12.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-08T08:32:12.428 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-08T08:32:12.472 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T08:32:12.473 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T08:32:12.809 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-08T08:32:12.809 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-03-08T08:32:12.809 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-08T08:32:12.829 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-03-08T08:32:12.830 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-08T08:32:12.849 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-08T08:32:12.849 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-08T08:32:12.872 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-08T08:32:12.918 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-08T08:32:12.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-08T08:32:12.941 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-08T08:32:13.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-08T08:32:13.069 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-08T08:32:13.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-08T08:32:13.096 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-08T08:32:13.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-08T08:32:13.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-08T08:32:13.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-08T08:32:13.161 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-03-08T08:32:13.189 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-03-08T08:32:13.205 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-08T08:32:13.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:32:13.229 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-08T08:32:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T08:32:13.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-08T08:32:13.268 INFO:teuthology.task.internal:Transferring archived files... 2024-03-08T08:32:13.269 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi012 2024-03-08T08:32:13.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-08T08:32:13.306 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510/remote/smithi156 2024-03-08T08:32:13.306 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-08T08:32:13.341 INFO:teuthology.task.internal:Removing archive directory... 2024-03-08T08:32:13.341 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-08T08:32:13.344 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-08T08:32:13.390 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-08T08:32:13.400 INFO:teuthology.task.internal:Not uploading archives. 2024-03-08T08:32:13.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-08T08:32:13.446 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-08T08:32:13.446 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-08T08:32:13.448 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-08T08:32:13.460 INFO:teuthology.orchestra.run.smithi012.stdout: 265361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 8 08:32 /home/ubuntu/cephtest 2024-03-08T08:32:13.462 INFO:teuthology.orchestra.run.smithi156.stdout: 266004 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 8 08:32 /home/ubuntu/cephtest 2024-03-08T08:32:13.463 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-08T08:32:13.473 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-08T08:32:13.574 INFO:teuthology.nuke:Checking targets against current locks 2024-03-08T08:32:13.602 DEBUG:teuthology.nuke:shortname: smithi012 2024-03-08T08:32:13.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T08:32:13.631 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 08:10:46.688364', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tSPXjRcxJrQoUq/rh0IHWWR+SpHIJgvsXRqThVIwRMImZoQVE9S+8dsvivc5X9Jd/QydIsWWroMlepiN1lIw='} 2024-03-08T08:32:13.632 DEBUG:teuthology.nuke:shortname: smithi156 2024-03-08T08:32:13.632 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T08:32:13.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2024-03-08_08:09:40-fs:functional-main-distro-default-smithi/7585510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 08:10:46.689304', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/z/uV83Z6d9tF1DAevK6aqUuf/yu+zfZfYdqeRUiZMOeCd3jl6UOBBo2DMw8+NT1UTHnp3ND54on+Yljk6URQ='} 2024-03-08T08:32:13.690 INFO:teuthology.orchestra.console.smithi012:Power off 2024-03-08T08:32:13.691 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T08:32:13.715 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-03-08T08:32:13.715 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:13.730 INFO:teuthology.orchestra.console.smithi156:Power off 2024-03-08T08:32:13.731 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T08:32:13.741 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-08T08:32:13.753 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-03-08T08:32:13.753 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:13.776 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-08T08:32:17.742 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:17.776 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:17.853 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-08T08:32:17.886 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-08T08:32:21.857 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:21.887 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T08:32:21.968 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-03-08T08:32:21.997 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-03-08T08:32:22.070 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-03-08T08:32:22.098 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-03-08T08:32:22.203 INFO:teuthology.run:Summary data: description: fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} duration: 755.3693289756775 failure_reason: 'Test failure: test_per_client_labeled_perf_counters (tasks.cephfs.test_admin.TestLabeledPerfCounters)' flavor: default owner: scheduled_jcollin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f88257a81fbd4caf8a2a4dc70d094848 status: fail success: false 2024-03-08T08:32:22.203 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T08:32:22.292 INFO:teuthology.run:FAIL